Image processing methods, face recognition methods, devices, equipment and storage media

Abstract

This application discloses an image processing method, a face recognition method, an apparatus, a device, and a storage medium, relating to the field of computer vision. The method includes: generating a color texture structure map based on a color image, and generating a depth texture structure map based on a depth image, wherein the color image and the depth map are two images of the same scene; obtaining the region structure weights of pixels in the depth texture structure map based on the adjacency relationships between pixels in the depth texture structure map and the positional correspondences between pixels in the depth texture structure map and pixels in the color texture structure map; and upsampling the depth map based on the region structure weights to obtain an upsampled depth map. This method can upsample the depth image, ensuring the resolution accuracy and field of view of the aligned image.

Description

Technical Field

[0001] This application relates to the field of computer vision, and in particular to an image processing method, a face recognition method, a device, an apparatus, and a storage medium. Background Technology

[0002] In facial recognition technology, color images, infrared images, and depth images of a face are used for facial recognition. Color images are used for face selection and comparison, depth images are used for liveness detection and to assist in comparison, and infrared images are used for liveness detection. During facial recognition, the color images, depth images, and infrared images need to be spatially aligned.

[0003] Because color and infrared images have high resolution, while depth images have low resolution, related technologies downsample the color and infrared images to the resolution of the depth image during three-image alignment to make the resolution of the three images consistent, thereby enabling three-image alignment.

[0004] The methods in related technologies that downsample color and infrared images will result in a loss of image resolution accuracy and field of view (FOV). Summary of the Invention

[0005] This application provides an image processing method, a face recognition method, an apparatus, a device, and a storage medium, which can upsample depth images to ensure the resolution accuracy and FOV of the aligned images. The technical solution is as follows.

[0006] According to one aspect of this application, an image processing method is provided, the method comprising:

[0007] A color texture structure map is generated based on a color image, and a depth texture structure map is generated based on a depth image. The color image and the depth image are two images of the same scene.

[0008] Based on the adjacency relationship between pixels in the depth texture structure map, and the positional correspondence between pixels in the depth texture structure map and pixels in the color texture structure map, the region structure weight of pixels in the depth map is obtained.

[0009] An upsampled depth map is obtained by upsampling the depth map based on the region structure weights.

[0010] In an optional embodiment, obtaining the region structure weights of pixels in the depth map based on the adjacency relationships between pixels in the depth texture structure map and the positional correspondence between pixels in the depth texture structure map and pixels in the color texture structure map includes:

[0011] The potential energy function is obtained by calculating the sum of the adjacency potential energy and the associated potential energy. The adjacency potential energy is calculated based on the adjacency weight of the pixels in the depth texture structure map, and the associated potential energy is calculated based on the associated weight of the pixels with the corresponding relationship in the depth texture structure map and the color texture structure map.

[0012] Adjust the adjacency weights and correlation weights during training so that the potential function satisfies the training stopping condition;

[0013] The adjacency weights at which the potential function reaches its minimum value are determined as the region structure weights of the pixels in the depth map.

[0014] In one optional embodiment, the depth texture structure map and the color texture structure map each include n pixels at the same position, where n is a positive integer;

[0015] The calculation of the sum of adjacent potential energy and associated potential energy to obtain the potential energy function includes:

[0016] Obtain n initial adjacency weights for the n pixels in the depth texture structure map; calculate the weighted sum of the Gaussian distribution function of the n pixels in the depth texture structure map and the adjacency weights to obtain the adjacency potential.

[0017] Obtain the n initial association weights corresponding to the n pixels between the depth texture structure map and the color texture structure map; calculate the weighted sum of the multivariate Gaussian distribution function of the n pixels in the depth texture structure map and the color texture structure map with the association weights to obtain the association potential energy;

[0018] The potential energy function is obtained by calculating the sum of the adjacent potential energy and the associated potential energy.

[0019] In one optional embodiment, the training stopping condition includes at least one of: stopping training when the potential function is less than a threshold, or stopping training when the number of training iterations reaches a threshold.

[0020] In an optional embodiment, the method further includes:

[0021] Repeat the steps of obtaining the region structure weights of pixels in the depth map based on the adjacency relationship between pixels in the depth texture structure map and the position correspondence between pixels in the depth texture structure map and pixels in the color texture structure map, to obtain at least two region structure weights of pixels in the depth map.

[0022] Calculate the weighted average of the weights of the at least two regional structures to obtain the weighted average of the regional structure weights;

[0023] The process of upsampling the depth map based on the region structure weights to obtain an upsampled depth map includes:

[0024] The upsampled depth map is obtained by upsampling the depth map based on the weighted average of the region structure weights.

[0025] In an optional embodiment, the method further includes:

[0026] The original color image is downsampled to obtain the color image, wherein the resolution of the original color image is higher than the resolution of the depth image, and the resolution of the color image is equal to the resolution of the depth image.

[0027] In one alternative embodiment, the depth map includes integer pixels, and the upsampled depth map includes the integer pixels and sub-pixels;

[0028] The process of upsampling the depth map based on the region structure weights to obtain an upsampled depth map includes:

[0029] The target depth value of the sub-pixel is calculated based on the coordinates of the sub-pixel, the depth values ​​of the four nearest integer pixels to the sub-pixel, and the region structure weights corresponding to the four nearest integer pixels to the sub-pixel.

[0030] The upsampled depth map of the depth image is generated based on the depth value of the integer pixel and the target depth value of the sub-pixel.

[0031] In one alternative embodiment, the region structure weights include three coefficients of a linear equation in two variables;

[0032] The calculation of the target depth value of the sub-pixel based on the coordinates of the sub-pixel, the depth values ​​of the four nearest integer pixels to the sub-pixel, and the region structure weights corresponding to the four nearest integer pixels to the sub-pixel includes:

[0033] Obtain the i-th coordinate of the sub-pixel relative to the i-th integer pixel among the four integer pixels, where i is a positive integer not greater than 4;

[0034] Substitute the i-th coordinate into the i-th region structure weight corresponding to the i-th integer pixel to obtain the i-th region structure weight value;

[0035] Calculate the product of the i-th region structure weight value and the depth value of the i-th integer pixel to obtain the i-th addend;

[0036] Repeat the above steps to obtain the four addends corresponding to the four integer pixels;

[0037] The sum of the four addends is determined as the target depth value of the sub-pixel.

[0038] According to another aspect of this application, a face recognition method is provided, the method comprising:

[0039] A color texture structure map is generated from the color map of the face image, and a depth texture structure map is generated from the depth map of the face image.

[0040] Based on the adjacency relationship between pixels in the depth texture structure map, and the positional correspondence between pixels in the depth texture structure map and pixels in the color texture structure map, the region structure weight of pixels in the depth map is obtained.

[0041] An upsampled depth map is obtained by upsampling the depth map based on the region structure weights.

[0042] Face recognition is performed based on the color image and the upsampled depth image.

[0043] In an optional embodiment, obtaining the region structure weights of pixels in the depth map based on the adjacency relationships between pixels in the depth texture structure map and the positional correspondence between pixels in the depth texture structure map and pixels in the color texture structure map includes:

[0044] The potential energy function is obtained by calculating the sum of the adjacency potential energy and the associated potential energy. The adjacency potential energy is calculated based on the adjacency weight of the pixels in the depth texture structure map, and the associated potential energy is calculated based on the associated weight of the pixels with the corresponding relationship in the depth texture structure map and the color texture structure map.

[0045] Adjust the adjacency weights and correlation weights during training so that the potential function satisfies the training stopping condition;

[0046] The adjacency weights at which the potential function reaches its minimum value are determined as the region structure weights of the pixels in the depth map.

[0047] In one optional embodiment, the depth texture structure map and the color texture structure map each include n pixels at the same position, where n is a positive integer;

[0048] The calculation of the sum of adjacent potential energy and associated potential energy to obtain the potential energy function includes:

[0049] Obtain n initial adjacency weights for the n pixels in the depth texture structure map; calculate the weighted sum of the Gaussian distribution function of the n pixels in the depth texture structure map and the adjacency weights to obtain the adjacency potential.

[0050] Obtain the n initial association weights corresponding to the n pixels between the depth texture structure map and the color texture structure map; calculate the weighted sum of the multivariate Gaussian distribution function of the n pixels in the depth texture structure map and the color texture structure map with the association weights to obtain the association potential energy;

[0051] The potential energy function is obtained by calculating the sum of the adjacent potential energy and the associated potential energy.

[0052] In one optional embodiment, the training stopping condition includes at least one of: stopping training when the potential function is less than a threshold, or stopping training when the number of training iterations reaches a threshold.

[0053] In an optional embodiment, the method further includes:

[0054] Repeat the steps of obtaining the region structure weights of pixels in the depth map based on the adjacency relationship between pixels in the depth texture structure map and the position correspondence between pixels in the depth texture structure map and pixels in the color texture structure map, to obtain at least two region structure weights of pixels in the depth map.

[0055] Calculate the weighted average of the weights of the at least two regional structures to obtain the weighted average of the regional structure weights;

[0056] The process of upsampling the depth map based on the region structure weights to obtain an upsampled depth map includes:

[0057] The upsampled depth map is obtained by upsampling the depth map based on the weighted average of the region structure weights.

[0058] In an optional embodiment, the method further includes:

[0059] The original color image is downsampled to obtain the color image, wherein the resolution of the original color image is higher than the resolution of the depth image, and the resolution of the color image is equal to the resolution of the depth image.

[0060] In one alternative embodiment, the depth map includes integer pixels, and the upsampled depth map includes the integer pixels and sub-pixels;

[0061] The process of upsampling the depth map based on the region structure weights to obtain an upsampled depth map includes:

[0062] The target depth value of the sub-pixel is calculated based on the coordinates of the sub-pixel, the depth values ​​of the four nearest integer pixels to the sub-pixel, and the region structure weights corresponding to the four nearest integer pixels to the sub-pixel.

[0063] The upsampled depth map of the depth image is generated based on the depth value of the integer pixel and the target depth value of the sub-pixel.

[0064] In one alternative embodiment, the region structure weights include three coefficients of a linear equation in two variables;

[0065] The calculation of the target depth value of the sub-pixel based on the coordinates of the sub-pixel, the depth values ​​of the four nearest integer pixels to the sub-pixel, and the region structure weights corresponding to the four nearest integer pixels to the sub-pixel includes:

[0066] Obtain the i-th coordinate of the sub-pixel relative to the i-th integer pixel among the four integer pixels, where i is a positive integer not greater than 4;

[0067] Substitute the i-th coordinate into the i-th region structure weight corresponding to the i-th integer pixel to obtain the i-th region structure weight value;

[0068] Calculate the product of the i-th region structure weight value and the depth value of the i-th integer pixel to obtain the i-th addend;

[0069] Repeat the above steps to obtain the four addends corresponding to the four integer pixels;

[0070] The sum of the four addends is determined as the target depth value of the sub-pixel.

[0071] According to another aspect of this application, an image processing apparatus is provided, the apparatus comprising:

[0072] The first texture module is used to generate a color texture structure map based on a color image and a depth texture structure map based on a depth image, wherein the color image and the depth image are two images of the same scene;

[0073] The first weighting module is used to obtain the region structure weight of the pixels in the depth map based on the adjacency relationship between pixels in the depth texture structure map and the positional correspondence between the pixels in the depth texture structure map and the pixels in the color texture structure map.

[0074] The first upsampling module is used to upsample the depth map based on the region structure weights to obtain an upsampled depth map.

[0075] In an optional embodiment, a first weighting module is used to calculate the sum of adjacent potential energy and associated potential energy to obtain a potential energy function. The adjacent potential energy is calculated based on the adjacent weights of pixels in the depth texture structure map, and the associated potential energy is calculated based on the associated weights of pixels with the corresponding relationship in the depth texture structure map and the color texture structure map.

[0076] The first weight module is used to adjust the adjacent weights and the associated weights during training so that the potential function satisfies the training stopping condition.

[0077] The first weighting module is used to determine the adjacency weight when the potential energy function reaches its minimum value as the region structure weight of the pixel in the depth map.

[0078] In one optional embodiment, the depth texture structure map and the color texture structure map each include n pixels at the same position, where n is a positive integer;

[0079] The first weighting module is used to obtain n initial adjacency weights of n pixels in the depth texture structure map; calculate the weighted sum of the Gaussian distribution function of the n pixels in the depth texture structure map and the adjacency weights to obtain the adjacency potential.

[0080] The first weighting module is used to obtain n initial association weights corresponding to the n pixels between the depth texture structure map and the color texture structure map; calculate the weighted sum of the multivariate Gaussian distribution function of the n pixels in the depth texture structure map and the color texture structure map and the association weights to obtain the association potential energy;

[0081] The first weighting module is used to calculate the sum of the adjacent potential energy and the associated potential energy to obtain the potential energy function.

[0082] In one optional embodiment, the training stopping condition includes at least one of: stopping training when the potential function is less than a threshold, or stopping training when the number of training iterations reaches a threshold.

[0083] In an optional embodiment, the first weighting module is used to repeatedly execute the step of obtaining the region structure weights of the pixels in the depth map based on the adjacency relationship between pixels in the depth texture structure map and the positional correspondence between the pixels in the depth texture structure map and the pixels in the color texture structure map, so as to obtain at least two region structure weights of the pixels in the depth map.

[0084] The first weighting module is used to calculate the weighted average of the weights of the at least two regional structures to obtain the weighted average of the regional structure weights.

[0085] The first upsampling module is used to upsample the depth map based on the weighted average of the regional structure weights to obtain the upsampled depth map.

[0086] In an optional embodiment, the device further includes:

[0087] The first downsampling module is used to downsample the original color image to obtain the color image, wherein the resolution of the original color image is higher than the resolution of the depth image, and the resolution of the color image is equal to the resolution of the depth image.

[0088] In one alternative embodiment, the depth map includes integer pixels, and the upsampled depth map includes the integer pixels and sub-pixels;

[0089] The first upsampling module is used to calculate the target depth value of the sub-pixel based on the coordinates of the sub-pixel, the depth values ​​of the four integer pixels closest to the sub-pixel, and the region structure weights corresponding to the four integer pixels closest to the sub-pixel.

[0090] The first upsampling module is used to generate the upsampled depth map of the depth image based on the depth value of the integer pixel and the target depth value of the sub-pixel.

[0091] In one alternative embodiment, the region structure weights include three coefficients of a linear equation in two variables;

[0092] The first upsampling module is used to obtain the i-th coordinate of the sub-pixel relative to the i-th integer pixel among the four integer pixels, where i is a positive integer not greater than 4;

[0093] The first upsampling module is used to substitute the i-th coordinate into the i-th region structure weight corresponding to the i-th integer pixel to obtain the i-th region structure weight value.

[0094] The first upsampling module is used to calculate the product of the i-th region structure weight value and the depth value of the i-th integer pixel to obtain the i-th addend;

[0095] The first upsampling module is used to repeat the above steps to obtain the four addends corresponding to the four integer pixels respectively;

[0096] The first upsampling module is used to determine the target depth value of the sub-pixel by summing the four addends.

[0097] According to another aspect of this application, a face recognition device is provided, the device comprising:

[0098] The second texture module is used to generate a color texture structure map based on the color map of the face image and to generate a depth texture structure map based on the depth map of the face image.

[0099] The second weighting module is used to obtain the region structure weight of the pixels in the depth map based on the adjacency relationship between pixels in the depth texture structure map and the positional correspondence between the pixels in the depth texture structure map and the pixels in the color texture structure map.

[0100] The second upsampling module is used to upsample the depth map based on the region structure weights to obtain an upsampled depth map.

[0101] The recognition module is used to perform face recognition based on the color image and the upsampled depth image.

[0102] In an optional embodiment, the second weighting module is used to calculate the sum of the adjacent potential energy and the associated potential energy to obtain a potential energy function. The adjacent potential energy is calculated based on the adjacent weights of pixels in the depth texture structure map, and the associated potential energy is calculated based on the associated weights of pixels with the corresponding relationship in the depth texture structure map and the color texture structure map.

[0103] The second weight module is used to adjust the adjacent weights and the associated weights during training so that the potential function satisfies the training stopping condition.

[0104] The second weighting module is used to determine the adjacency weight when the potential energy function reaches its minimum value as the region structure weight of the pixel in the depth map.

[0105] In one optional embodiment, the depth texture structure map and the color texture structure map each include n pixels at the same position, where n is a positive integer;

[0106] The second weighting module is used to obtain the n initial adjacency weights of the n pixels in the depth texture structure map; calculate the weighted sum of the Gaussian distribution function of the n pixels in the depth texture structure map and the adjacency weights to obtain the adjacency potential.

[0107] The second weighting module is used to obtain n initial association weights corresponding to the n pixels between the depth texture structure map and the color texture structure map; calculate the weighted sum of the multivariate Gaussian distribution function of the n pixels in the depth texture structure map and the color texture structure map and the association weights to obtain the association potential energy;

[0108] The second weighting module is used to calculate the sum of the adjacent potential energy and the associated potential energy to obtain the potential energy function.

[0109] In one optional embodiment, the training stopping condition includes at least one of: stopping training when the potential function is less than a threshold, or stopping training when the number of training iterations reaches a threshold.

[0110] In an optional embodiment, the second weighting module is used to repeatedly execute the step of obtaining the region structure weights of the pixels in the depth map based on the adjacency relationship between pixels in the depth texture structure map and the positional correspondence between the pixels in the depth texture structure map and the pixels in the color texture structure map, so as to obtain at least two region structure weights of the pixels in the depth map.

[0111] The second weighting module is used to calculate the weighted average of the weights of the at least two regional structures to obtain the weighted average of the regional structure weights.

[0112] The second upsampling module is used to upsample the depth map based on the weighted average of the regional structure weights to obtain the upsampled depth map.

[0113] In an optional embodiment, the device further includes:

[0114] The second downsampling module is used to downsample the original color image to obtain the color image, wherein the resolution of the original color image is higher than the resolution of the depth image, and the resolution of the color image is equal to the resolution of the depth image.

[0115] In one alternative embodiment, the depth map includes integer pixels, and the upsampled depth map includes the integer pixels and sub-pixels;

[0116] The second upsampling module is used to calculate the target depth value of the sub-pixel based on the coordinates of the sub-pixel, the depth values ​​of the four integer pixels closest to the sub-pixel, and the region structure weights corresponding to the four integer pixels closest to the sub-pixel.

[0117] The second upsampling module is used to generate the upsampled depth map of the depth image based on the depth value of the integer pixel and the target depth value of the sub-pixel.

[0118] In one alternative embodiment, the region structure weights include three coefficients of a linear equation in two variables;

[0119] The second upsampling module is used to obtain the i-th coordinate of the sub-pixel relative to the i-th integer pixel among the four integer pixels, where i is a positive integer not greater than 4;

[0120] The second upsampling module is used to substitute the i-th coordinate into the i-th region structure weight corresponding to the i-th integer pixel to obtain the i-th region structure weight value.

[0121] The second upsampling module is used to calculate the product of the i-th region structure weight value and the depth value of the i-th integer pixel to obtain the i-th addend;

[0122] The second upsampling module is used to repeat the above steps to obtain the four addends corresponding to the four integer pixels respectively;

[0123] The second upsampling module is used to determine the target depth value of the sub-pixel by summing the four addends.

[0124] According to another aspect of this application, a computer device is provided, the computer device comprising: a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the image processing method and / or face recognition method as described above.

[0125] According to another aspect of this application, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the image processing method and / or face recognition method as described above.

[0126] According to another aspect of the embodiments of this disclosure, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the image processing method and / or face recognition method provided in the above-described optional implementations.

[0127] The beneficial effects of the technical solutions provided in this application include at least the following:

[0128] By using texture maps from both the color image and depth map, and leveraging the shared texture patterns and the correlation between adjacent pixels in the depth map, the region structure weights of each pixel in the depth map are calculated. These weights are then used to upsample the depth map. In scenarios where the color image has high resolution but the depth map has low resolution, the depth map can be upsampled to the same resolution as the color image. This adapts the depth map to the color image, reducing resolution loss after image alignment, improving the field of view (FOV), and ultimately increasing the resolution of the aligned image. Consequently, this improves the accuracy of image recognition and face recognition when using the aligned image. Attached Figure Description

[0129] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0130] Figure 1 This is a block diagram of a computer device provided in an exemplary embodiment of this application;

[0131] Figure 2 This is a flowchart of an image processing method provided in another exemplary embodiment of this application;

[0132] Figure 3 This is a flowchart of an image processing method provided in another exemplary embodiment of this application;

[0133] Figure 4 This is a schematic diagram of an image processing method provided in another exemplary embodiment of this application;

[0134] Figure 5 This is a schematic diagram of an image processing method provided in another exemplary embodiment of this application;

[0135] Figure 6 This is a schematic diagram of an image processing method provided in another exemplary embodiment of this application;

[0136] Figure 7 This is a schematic diagram of an image processing method provided in another exemplary embodiment of this application;

[0137] Figure 8 This is a flowchart of a face recognition method provided in another exemplary embodiment of this application;

[0138] Figure 9 This is a block diagram of an image processing apparatus provided in another exemplary embodiment of this application;

[0139] Figure 10 This is a block diagram of a face recognition device provided in another exemplary embodiment of this application;

[0140] Figure 11 This is a schematic diagram of the server structure provided in another exemplary embodiment of this application;

[0141] Figure 12 This is a block diagram of a terminal provided in another exemplary embodiment of this application. Detailed Implementation

[0142] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0143] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0144] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0145] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing, tracking, and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0146] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learn-by-doing.

[0147] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0148] Figure 1 A schematic diagram of a computer device 101 provided in an exemplary embodiment of this application is shown. The computer device 101 may be a terminal or a server.

[0149] The terminal may include at least one of a digital camera, smartphone, laptop, desktop computer, tablet, smart speaker, and smart robot. Optionally, the terminal may also be a device with a camera, such as a facial recognition payment device, monitoring device, or access control device. In one optional implementation, the image processing method provided in this application can be applied to an application with image processing capabilities. This application may be a camera application, image processing application, video processing application, image publishing application, video publishing application, social application, shopping application, live streaming application, forum application, news application, lifestyle application, or office application. Optionally, a client application of this application is installed on the terminal.

[0150] For example, the terminal stores an image processing algorithm 102. When the client needs to use the image processing function, the client can call the image processing algorithm to complete the image processing. For example, the image processing process can be completed by the terminal or by the server.

[0151] The terminal and the server are connected to each other via wired or wireless network.

[0152] The terminal includes a first memory and a first processor. The first memory stores image processing algorithms; these image processing algorithms are invoked and executed by the first processor to implement the image processing method or face recognition method provided in this application. The first memory may include, but is not limited to, the following: Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EEPROM).

[0153] The first processor can consist of one or more integrated circuit chips. Optionally, the first processor can be a general-purpose processor, such as a central processing unit (CPU) or a network processor (NP). Optionally, the first processor can implement the image processing method or face recognition method provided in this application by running programs or code.

[0154] The server includes a second memory and a second processor. The second memory stores image processing algorithms; these image processing algorithms are invoked by the second processor to implement the image processing method or face recognition method provided in this application. Optionally, the second memory may include, but is not limited to, the following: RAM, ROM, PROM, EPROM, and EEPROM. Optionally, the second processor may be a general-purpose processor, such as a CPU or NP.

[0155] like Figure 1 As shown, computer device 101 calls image processing algorithm 102 to upsample the depth map based on the depth map and color map to obtain an upsampled depth map. Computer device 101 obtains a depth texture structure map from the depth map and a color texture structure map from the color map. Utilizing the principle of Markov chains, based on the correlation between adjacent pixels in the depth texture structure map and the correlation between pixels at the same position in the depth texture structure map and the color texture structure map, it calls a region structure weight model to calculate the region structure weight of each pixel in the depth map. Then, based on the calculated region structure weights, it performs upsampling interpolation on the depth map, interpolating between the original integer pixels to obtain the pixel values ​​of sub-pixels, thus obtaining the upsampled depth map.

[0156] Figure 2 A flowchart illustrating an exemplary embodiment of the image processing method provided in this application is shown. This method can be executed by a computer device, such as... Figure 1 The method is executed using the terminal or server shown. The method includes the following steps.

[0157] Step 201: Generate a color texture structure map based on the color image and a depth texture structure map based on the depth image. The color image and the depth image are two images of the same scene.

[0158] Color images and depth maps are two types of images from the same image. For example, a computer device uses an image acquisition device to capture the same scene, obtaining a color image (original color image) and a depth map of that scene. Because the acquisition precision of the two types of images differs, the original color image and the depth map have different resolutions. To better utilize the two images in subsequent processing, it is necessary to align them to the same resolution. The method provided in this embodiment can upsample the depth map, increasing its resolution, and thus improving the resolution of the aligned image.

[0159] For example, a computer device downsamples the original color image to obtain a color image. The resolution of the original color image is higher than that of the depth image, and the resolution of the color image is equal to that of the depth image. The color image and the depth image in step 201 have the same resolution (sharpness).

[0160] Color images and depth images can be images captured in real time by computer devices, images stored locally by computer devices, or images read or received by computer devices from other devices.

[0161] A texture structure map is an image obtained by mapping a raw image to a texture map. Texture structure maps are used to characterize the texture properties of objects. The texture mapping methods used to generate texture structure maps can employ spatial domain methods: using the spatial adjacency of the image as the computational field, and employing methods such as histogram analysis or autocorrelation functions to extract the texture features of the image.

[0162] The method for extracting image texture features to generate texture structure maps can be arbitrary. For example, texture structure map generation methods can include: texture feature analysis of gray-level co-occurrence matrix, extraction of texture features using image autocorrelation function, semivariogram method, Voronio checkerboard feature method, Markov random field model method, Gibbs random field model method, fractal model and autoregressive texture model, etc.

[0163] The computer device extracts texture features from the depth map and the color image respectively, resulting in a depth texture structure map and a color texture structure map. The depth map, color image, depth texture structure map, and color texture structure map are all spatial domain images with the same resolution and a one-to-one correspondence between their pixels. For example, if all four images contain n pixels, the i-th pixel in the depth map, the i-th pixel in the color image, the i-th pixel in the depth texture structure map, and the i-th pixel in the color texture structure map have a corresponding relationship.

[0164] Step 202: Based on the adjacency relationship between pixels in the depth texture structure map and the positional correspondence between pixels in the depth texture structure map and pixels in the color texture structure map, obtain the region structure weight of pixels in the depth map.

[0165] In a depth texture structure diagram, two adjacent pixels are adjacent to each other.

[0166] Two pixels at the same position in the depth texture structure map and the color texture structure map have a positional correspondence.

[0167] Based on the constraints of the above two relationships, the region structure weights of each pixel (pixel position) in the depth texture structure map (depth map) are calculated. The depth map is then upsampled based on the region structure weights to obtain the upsampled depth map.

[0168] Region structure weights are used to characterize the numerical variation relationship of pixel values ​​among individual pixels in the depth texture structure map, as well as the numerical variation relationship of pixel values ​​at the same pixel position between the depth texture structure map and the color texture structure map. Since the numerical variation trend of pixel values ​​at the same position in the depth texture structure map and the depth map is the same, upsampling the depth map based on the obtained region structure weights can improve the accuracy of the depth values ​​of the upsampled pixels (sub-pixels).

[0169] For example, region structure weights are the weights of each pixel position in the depth map or depth texture structure map. The same pixel position in the depth map and depth texture structure map corresponds to the same region structure weight.

[0170] A Markov chain is a stochastic process that possesses Markov properties and exists within a discrete set of exponents and a state space. It can be defined using a transition matrix and a transition graph. In this embodiment, it is used to derive the coupling relationship between two texture structure graphs, thereby determining the upsampling weights of the depth map.

[0171] Region structure weights refer to weights that can simultaneously reflect the texture structure of the depth map and the color map, thereby ensuring that during upsampling, the depth map retains the original edge characteristics of the depth map (3D edges) and the color map (2D edges).

[0172] Step 203: Upsample the depth map based on the region structure weights to obtain an upsampled depth map.

[0173] Based on the obtained region structure weights of each pixel location and the depth values ​​of each pixel location in the depth map, the depth map is upsampled and interpolated to obtain an upsampled depth map.

[0174] Upsampling interpolation refers to the process of increasing the resolution of a depth map by upsampling it. The pixel values ​​(depth values) of the extra pixels (sub-pixels) compared to the original depth map are obtained by weighting the region structure weights and the original pixel values ​​(integers) to get the pixel values ​​of the extra pixels, thus producing the upsampled depth map.

[0175] In summary, the method provided in this embodiment uses texture maps of both a color image and a depth map. Based on the characteristic that the color image and depth map share the same texture pattern, and the correlation between adjacent pixels in the depth map, it calculates the region structure weights of each pixel in the depth map. These region structure weights are then used to upsample the depth map, thereby achieving upsampling. In scenarios where the color image has a high resolution and the depth map has a low resolution, the depth map can be upsampled to the same resolution as the color image. This adapts the depth map to the color image, reducing resolution loss after image alignment, increasing the field of view (FOV), and improving the resolution of the aligned image. Consequently, it improves the accuracy of image recognition and face recognition when using the aligned image.

[0176] An exemplary embodiment for calculating the weights of a region structure is provided.

[0177] Figure 3 A flowchart illustrating an exemplary embodiment of the image processing method provided in this application is shown. This method can be executed by a computer device, such as... Figure 1 The method is executed using the terminal or server shown. The method includes the following steps.

[0178] Step 201: Generate a color texture structure map based on the color image and a depth texture structure map based on the depth image. The color image and the depth image are two images of the same scene.

[0179] For example, prior to step 201, the computer device downsamples the original color image to obtain a color image, wherein the resolution of the original color image is higher than the resolution of the depth image, and the resolution of the color image is equal to the resolution of the depth image.

[0180] Step 2021: Calculate the sum of the adjacency potential and the associated potential to obtain the potential function. The adjacency potential is calculated based on the adjacency weight of the pixels in the depth texture structure map, and the associated potential is calculated based on the associated weight of the corresponding pixels in the depth texture structure map and the color texture structure map.

[0181] Taking a depth map and color map comprising m pixels, and an upsampled depth map comprising n pixels as an example, where m is less than n. The m pixels in the depth texture structure map and color texture structure map are upsampled to n pixels. For example, upsampling one pixel in the original depth texture structure map to 9*9 pixels results in the pixel value of the upsampled 9*9 pixels being equal to the pixel value of that one pixel in the original depth texture structure map. The pixel at the center of the 9*9 pixels is called the integer pixel (the original pixel), and the other 80 pixels generated by the upsampling interpolation are called sub-pixels (the newly generated pixels).

[0182] For example, such as Figure 4 As shown, the original pixel 501 in the depth texture structure map and color texture structure map is upsampled to 9*9 pixels. The pixel values ​​of the 9*9 pixels after upsampling are the same as the pixel values ​​of pixel 501. The pixel at the center of the 9*9 pixels is called the integer pixel 502, and the other 80 pixels are called sub-pixels.

[0183] Thus, as Figure 5 As shown, taking a depth map and a color map that include 4 pixels as an example, after upsampling, the depth texture structure map can include 4*9*9 pixels, so there are 8 sub-pixel points between every two whole pixels.

[0184] Each sub-pixel has an adjacency weight corresponding to one of its four nearest integer pixels; that is, a sub-pixel has four adjacency weights. Integer pixels have an adjacency weight of 0.

[0185] The depth texture structure map and the color texture structure map each include n pixels at the same position, where n is a positive integer. The computer device obtains n initial adjacency weights for the n pixels in the depth texture structure map; calculates the weighted sum of the Gaussian distribution function of the n pixels in the depth texture structure map and the adjacency weights to obtain the adjacency potential; obtains n initial association weights for the n pixels corresponding to the depth texture structure map and the color texture structure map; calculates the weighted sum of the multivariate Gaussian distribution function of the n pixels in the depth texture structure map and the color texture structure map and the association weights to obtain the association potential; calculates the sum of the adjacency potential and the association potential to obtain the potential energy function.

[0186] The formula for the potential energy function is as follows:

[0187] E(x)=-logp(y|x)-logp(x)=∑ p∈V θ p (x p )+∑ (p，q)∈E θ pq (x p ,x q ).

[0188] Where E(x) is the potential energy function, p represents the depth map (depth texture structure map), q represents the color map (color texture structure map), and p and q represent the same position in the depth texture structure map and the color texture structure map.

[0189] V includes n pixels (including integer pixels and sub-pixels) obtained by upsampling the depth texture structure map, and E includes n pairs of corresponding pixels (including integer pixels and sub-pixels) obtained by upsampling the depth texture structure map and the color texture structure map.

[0190] x p x is the pixel value of a pixel in the depth texture structure map. q For the color texture structure diagram and x p The pixel value of the pixel at the corresponding pixel position.

[0191] Where, θ p (x p )=f(x p )*w p ;

[0192]

[0193] w p=w p1 +w p2 +w p3 +w p4 =(a p1 x p1 +b p1 y p1 +c p1 )+(a p2 x p2 +bp2yp2+cp2+ap3xp3+bp3yp3+cp3+ap4xp4+bp4yp4+cp4;

[0194] f(x p ) is x p The multivariate Gaussian distribution function, μ p σ is the average value of all pixel values ​​in the depth texture structure map. p It represents the variance of all pixel values ​​in the depth texture structure map.

[0195] w p For x p The adjacency weight, w p1 For x p Compared to the adjacency weight of the first integer pixel among four adjacent integer pixels; w p2 For x p Compared to the adjacency weight of the second integer pixel out of four adjacent integer pixels; w p3 For x p Compared to the adjacency weight of the third integer pixel out of four adjacent integer pixels; w p4 For x p The adjacency weight relative to the fourth integer pixel among the four adjacent integer pixels.

[0196] a p1 b p1 c p1 For x p Compared to the adjacency weight of the first integer pixel among four adjacent integer pixels, x p1 y p1 For x p The coordinate distance relative to the first integer pixel among four adjacent integer pixels; a p2 b p2 c p2 For x p Compared to the adjacency weight of the second integer pixel out of four adjacent integer pixels, x p2 y p2 For x p The coordinate distance relative to the second integer pixel among four adjacent integer pixels; a p3 b p3 cp3 For x p Compared to the adjacency weight of the third integer pixel out of four adjacent integer pixels, x p3 y p3 For x p The coordinate distance relative to the third integer pixel among four adjacent integer pixels; a p4 b p4 c p4 For x p Compared to the adjacency weight of the fourth integer pixel out of four adjacent integer pixels, x p4 y p4 For x p The coordinate distance relative to the fourth integer pixel out of four adjacent integer pixels. p1 y p1 x p2 y p2 x p3 y p3 x p4 y p4 The value range is [0 / 8, 8 / 8].

[0197] θ pq (x p ,x q )=f(x p ,x q )*w pq ;

[0198]

[0199] w pq =(a pq x p1 +b pq y p1 +c pq )+(a pq x p2 +b pq y p2 +c pq )+(a pq x p3 +bpqyp3+cpp+apqxp4+bpqyp4+cpq;

[0200] f(x p ,x q ) is x p and x q The multivariate Gaussian distribution function, μ p σ is the average value of all pixel values ​​in the depth texture structure map. p μ is the variance of all pixel values ​​in the depth texture structure map. qσ is the average value of all pixel values ​​in the color texture structure map. q This represents the variance of all pixel values ​​in the color texture structure map.

[0201] w pq For x p and x q The association weight, a pq b pq c pq For x p and x q The value of the association weight; x p1 y p1 For x p The distance x is relative to the coordinates of the first integer pixel among four adjacent integer pixels. p2 y p2 For x p The coordinate distance relative to the second integer pixel among four adjacent integer pixels; x p3 y p3 For x p The distance x is relative to the coordinates of the third integer pixel out of four adjacent integer pixels. p4 y p4 For x p The coordinate distance relative to the fourth integer pixel among four adjacent integer pixels.

[0202] In summary, each subpixel corresponds to four adjacency weights and one association weight. The four adjacency weights correspond to the four integer pixels adjacent to that subpixel. During the training phase, the computer needs to train and adjust the coefficients a, b, and c of these five weights (four adjacency weights and one association weight) to minimize the potential function E(x). When the potential function E(x) is minimized, the final four adjacency weights and one association weight corresponding to each subpixel in the depth texture structure map can be obtained. These four adjacency weights are the region structure weights of that subpixel. Based on these region structure weights and the pixel values ​​(depth values) of the four adjacent integer pixels, the pixel value (depth value) of the subpixel is calculated, achieving accurate upsampling.

[0203] For example, the adjacent potential energy in step 2021 refers to ∑ in the above formula. p∈V θ p (x p The related potential energy in step 2021 refers to the ∑ in the above formula. (p,q)∈E θ pq (x p ,x q ).

[0204] like Figure 6As shown, taking sub-pixel P in the depth texture structure map as an example, the four integer pixels closest to sub-pixel P are: the first integer pixel 503 located in the upper left corner, the second integer pixel 504 located in the upper right corner, the third integer pixel 505 located in the lower right corner, and the fourth integer pixel 506 located in the lower right corner. The coordinate distance of sub-pixel P relative to the first integer pixel 503 is (7 / 8, 7 / 8), the coordinate distance of sub-pixel P relative to the second integer pixel 504 is (2 / 8, 7 / 8), the coordinate distance of sub-pixel P relative to the third integer pixel 505 is (7 / 8, 2 / 8), and the coordinate distance of sub-pixel P relative to the fourth integer pixel 506 is (2 / 8, 2 / 8). The pixel value of sub-pixel P in the depth texture structure map is the same as the pixel value of the fourth integer pixel 506. Subpixel point P has four adjacent weights corresponding to the four integer pixels, and one association weight. During the training process, the computer device adjusts the values ​​of the five weights to minimize the potential function, and finally obtains the final values ​​of the five weights of the final point P.

[0205] Optionally, the depth texture structure map and the color texture structure map include m pixels. The computer device upsamples the depth texture structure map and the color texture structure map to obtain the upsampled depth texture structure map and color texture structure map. Each pixel is upsampled to x*x pixels, where x is a positive integer. The pixel value of the x*x pixels is taken as the pixel value of the pixel in the depth texture structure map or the color texture structure map. One pixel among the x*x pixels is determined as an integer pixel, and the (x*x-1) pixels other than the integer pixel are determined as sub-pixels. The upsampled depth texture structure map and color texture structure map each include n pixels at the same position, n = m*x*x; the computer device obtains n initial adjacency weights for the n pixels in the upsampled depth texture structure map; the Gaussian distribution function of the n pixels in the upsampled depth texture structure map is calculated as a weighted sum of the adjacency weights to obtain the adjacency potential; n initial association weights for the n pixels between the upsampled depth texture structure map and the upsampled color texture structure map are obtained; the multivariate Gaussian distribution function of the n pixels in the upsampled depth texture structure map and the upsampled color texture structure map is calculated as a weighted sum of the association weights to obtain the association potential; the sum of the adjacency potential and the association potential is calculated to obtain the potential energy function.

[0206] Step 2022: Adjust the training adjacency weights and correlation weights so that the potential function satisfies the training stopping condition.

[0207] For example, the computer device can randomly adjust the adjacency weight and association weight of each sub-pixel, or adjust the adjacency weight and association weight of each sub-pixel according to a certain numerical adjustment direction and step, or adjust some parameters of the adjacency weight and association weight according to a certain adjustment strategy.

[0208] For example, keep the values ​​of b and c in the fixed adjacency weight and association weight unchanged, and adjust the value of a.

[0209] Each time the computer device adjusts the adjacency weights and association weights, it calculates the potential function. Through iterative training and adjustment, multiple potential functions can be obtained. Training stops when the potential function meets the training stopping condition.

[0210] Training termination conditions may include at least one of the following: stopping training when the potential function is less than a threshold, or stopping training when the number of training iterations reaches a threshold.

[0211] That is, when the value of the potential energy function is less than the preset value, the adjustment of the adjacency weight and the association weight is stopped, and the adjacency weight at this time is used as the final region structure weight.

[0212] Alternatively, after obtaining a preset number of potential energy function values, stop adjusting the adjacency weights and association weights, find the minimum value of the potential energy function from the obtained multiple potential energy functions, and use the adjacency weight at the minimum value as the final region structure weight.

[0213] Step 2023: Determine the adjacency weights of the potential energy function when it reaches its minimum value as the region structure weights of pixels in the depth map.

[0214] For example, the adjacency weights when the potential function reaches its minimum value are used as the region structure weights for each sub-pixel (the pixel location where the sub-pixel is located). That is, each sub-pixel corresponds to four sets of region structure weights, and each set of region structure weights includes a set of abc coefficients. In other words, each sub-pixel obtains four sets of abc coefficients, and each set of abc coefficients corresponds to one of the four integer pixels.

[0215] For example, after a computer device obtains the region structure weights of a depth map through one training iteration, it can input the same depth map into the algorithm again to obtain more region structure weights. This process can be repeated multiple times to obtain multiple region structure weights. The computer device can then perform a weighted average of these multiple region structure weights to obtain the final region structure weights. The weighting coefficient for each region structure weight can be arbitrarily set. For example, the weighting coefficient for the first obtained region structure weight can be set to a slightly larger value, such as 0.8, while the weighting coefficients for subsequent obtained region structure weights can be set to slightly smaller values, such as 0.1 or 0.05. Then, the weighted average of the multiple region structure weights obtained through training can be performed to obtain the final region structure weights.

[0216] For example, the computer device repeatedly performs the steps of obtaining the region structure weights of pixels in the depth map based on the adjacency relationship between pixels in the depth texture structure map and the positional correspondence between pixels in the depth texture structure map and pixels in the color texture structure map, thereby obtaining at least two region structure weights of pixels in the depth map; calculating the weighted average of the at least two region structure weights to obtain the weighted averaged region structure weights; and upsampling the depth map based on the weighted averaged region structure weights to obtain an upsampled depth map.

[0217] For example, for a first depth map and a first color map, the computer device inputs them into a region structure weight model three times to obtain three region structure weights for the first depth map. Each region structure weight of the first depth map includes four sets of region structure weights corresponding to each sub-pixel in the first depth map. The region structure weight model is used to perform the step of obtaining the region structure weights of pixels in the depth map based on the adjacency relationship between pixels in the depth texture structure map and the positional correspondence between pixels in the depth texture structure map and pixels in the color texture structure map. Taking the calculation results of the first set of region structure weights corresponding to the first sub-pixel in the first depth map in the three calculations as follows: a=1, b=1, c=1; a=2, b=2, c=2; a=3, b=3, c=3 as an example, the following explanation will be provided. Assuming the weighting coefficient for the first region structure weight is 0.8, and the weighting coefficients for the second and third weighting coefficients are 0.1, then after weighted averaging, the final values ​​of the first group of region structure weights corresponding to the first sub-pixel in the first depth map are: a = 0.8*1 + 2*0.1 + 3*0.1 = 1.3, b = 0.8*1 + 2*0.1 + 3*0.1 = 1.3, c = 0.8*1 + 2*0.1 + 3*0.1 = 1.3.

[0218] Step 2031: Calculate the target depth value of the sub-pixel based on the coordinates of the sub-pixel, the depth values ​​of the four nearest integer pixels to the sub-pixel, and the region structure weights corresponding to the four nearest integer pixels to the sub-pixel.

[0219] The pixels in the original depth map are called integer pixels. Upsampling the depth map involves inserting new sub-pixels between the integer pixels; for example, inserting eight sub-pixels between every two integer pixels. The resulting upsampled depth map includes both integer and sub-pixels.

[0220] For example, such as Figure 7 As shown, the original depth map includes four pixels: the first integer pixel 503, the second integer pixel 504, the third integer pixel 505, and the fourth integer pixel 506. The original depth map is upsampled by inserting eight sub-pixels between every two integer pixels to obtain an upsampled depth map of 10*10 pixels after interpolation and upsampling.

[0221] For example, it can also be understood that the integer pixels in the original depth map are divided into x*x pixels, the pixel in the middle position of the x*x pixels is taken as the new integer pixel, and the other (x*x-1) pixels are called sub-pixels. Then the upsampled depth map includes the new integer pixels and sub-pixels.

[0222] For example, such as Figure 4 As shown, the original depth map includes one pixel: pixel 501 (an integer pixel). Upsampling the original depth map splits the pixel in the depth map into 9*9 pixels, resulting in an upsampled depth map of 9*9 pixels, which includes a new integer pixel 502 and 80 sub-pixels.

[0223] For example, the region structure weights include three coefficients (a, b, c) of a linear equation in two variables (ax + by + c). The computer device obtains the i-th coordinate of the sub-pixel relative to the i-th integer pixel among the four integer pixels, where i is a positive integer not greater than 4. The i-th coordinate is substituted into the i-th region structure weight corresponding to the i-th integer pixel to obtain the i-th region structure weight value. The product of the i-th region structure weight value and the depth value of the i-th integer pixel is calculated to obtain the i-th addend. The above steps are repeated to obtain the four addends corresponding to the four integer pixels respectively. The sum of the four addends is determined as the target depth value of the sub-pixel.

[0224] For example, such as Figure 6 As shown, taking four points on the depth map—the first integer pixel 503, the second integer pixel 504, the third integer pixel 505, and the fourth integer pixel 506—as an example, the target depth value of the sub-pixel point P is interpolated using the following formula:

[0225] P=Q1*W1+Q2*W2+Q3*W3+Q4*W4=Q1*(a1*x1+b1*y1+c1)+Q2*(a2*x2+b2*y2+c2)+Q3*(a3*x3+b3*y3+c3)+Q1*(a4*x4+b4*y4+c4).

[0226] Wherein, P is the target depth value of subpixel P, Q1, Q2, Q3, and Q4 are the depth values ​​of the first integer pixel 503, the second integer pixel 504, the third integer pixel 505, and the fourth integer pixel 506, respectively, and W1, W2, W3, and W4 are the region structure weights corresponding to subpixel P and the first integer pixel 503, the second integer pixel 504, the third integer pixel 505, and the fourth integer pixel 506, respectively. a1, b1, and c1 are the values ​​of the region structure weights corresponding to subpixel P and the first integer pixel 503, a2, b2, and c2 are the values ​​of the region structure weights corresponding to subpixel P and the second integer pixel 504, a3, b3, and c3 are the values ​​of the region structure weights corresponding to subpixel P and the third integer pixel 505, and a4, b4, and c4 are the values ​​of the region structure weights corresponding to subpixel P and the fourth integer pixel 506. x1 and y1 are the distance coordinates between sub-pixel point P and the first integer pixel point 503, with values ​​of (7 / 8, 7 / 8); x2 and y2 are the distance coordinates between sub-pixel point P and the second integer pixel point 504, with values ​​of (2 / 8, 7 / 8); x3 and y3 are the distance coordinates between sub-pixel point P and the third integer pixel point 505, with values ​​of (7 / 8, 2 / 8); x4 and y4 are the distance coordinates between sub-pixel point P and the fourth integer pixel point 506, with values ​​of (2 / 8, 2 / 8).

[0227] Step 2032: Generate an upsampled depth map of the depth image based on the depth values ​​of integer pixels and the target depth values ​​of sub-pixels.

[0228] After obtaining the target depth value of the newly inserted sub-pixel, the original depth map can be upsampled by combining it with the depth value of the integer pixel in the original depth map, thus obtaining the upsampled depth map.

[0229] In summary, the method provided in this embodiment uses texture maps of both a color image and a depth map, calculates the region structure weights of the depth map based on the two texture maps, and then uses these region structure weights to upsample the depth map. This upsampling of the depth map is achieved in scenarios where the color image has a high resolution and the depth map has a low resolution. The depth map can be upsampled to the same resolution as the color image to adapt it to the color image, reducing resolution loss after image alignment, increasing the field of view (FOV), and improving the resolution of the aligned image. This, in turn, improves the accuracy of image recognition and face recognition when using the aligned image.

[0230] The method provided in this embodiment uses a region-adaptive weighting model to characterize the spatial and textural consistency between depth and color images. This model couples depth and color information in different ways, achieving different weight ratios, by judging the characteristics of local regions in the depth image. The weights are used for weighted operations during upsampling interpolation.

[0231] An exemplary embodiment of applying the image processing method provided in this application in a face recognition scenario is provided.

[0232] Figure 8 A flowchart illustrating an exemplary embodiment of the image processing method provided in this application is shown. This method can be executed by a computer device, such as... Figure 1 The method is executed using the terminal or server shown. The method includes the following steps.

[0233] Step 301: Generate a color texture structure map from the color image of the face image, and generate a depth texture structure map from the depth image of the face image.

[0234] For example, the color image and the depth image correspond to the same face image.

[0235] The methods of steps 301 to 303 in this embodiment can be referred to Figure 2 Steps 201 to 203 of the illustrated embodiment, or, refer to... Figure 3 Steps 201 to 2032 of the illustrated embodiment. This embodiment will... Figure 2 and Figure 3 The illustrated embodiment is applied to a scenario where payment is made via facial recognition (face payment).

[0236] In one alternative implementation, the computer device captures color, depth, and infrared images of a face in real time for face recognition.

[0237] A color image refers to a color image captured by a color sensor under natural light. In facial recognition scenarios or payment scenarios using facial recognition, it is generally used for: face selection and comparison recognition.

[0238] A depth map is a map obtained by capturing infrared light with a speckle structure using an infrared sensor and then resolving the speckle using depth cells. In 3D computer graphics and computer vision, a depth map is an image or image channel containing information about the distance from the surface of a scene object to the viewpoint. Each pixel in the depth map represents the vertical distance between the depth camera plane and the plane of the object being photographed, typically represented by 16 bits and measured in millimeters. In facial recognition scenarios or payment scenarios using facial recognition, it is generally used for: liveness detection and assisting in comparative recognition.

[0239] Infrared images are infrared images captured by infrared sensors that capture infrared light. In facial recognition scenarios or payment scenarios using facial recognition, they are generally used for: liveness detection.

[0240] Face selection refers to choosing a set of color images, depth images, and infrared images that meet the prerequisites for liveness detection and comparison recognition algorithms. Selection is achieved by optimizing the color image based on face angle, face size, face centering, and clarity; optimizing the infrared image based on brightness; and optimizing the depth image based on completeness.

[0241] The preferred payment frame refers to selecting a set of color images, depth images, and infrared images from multiple sets of color images, depth images, and infrared images for use in liveness detection and comparison recognition.

[0242] Liveness detection refers to the detection method that determines whether the person using facial recognition is a real person, a photo, or a head model. Generally, depth maps are used to determine whether it is a photo, and the brightness of infrared images is used to determine whether it is a silicone head model.

[0243] Assisted comparison recognition refers to comparing and identifying which user is being scanned by facial recognition. It typically involves extracting five-point special similarity from a color image and then using a depth image to assist in comparing the three-dimensional similarity of these five points.

[0244] After acquiring the color image, depth image, and infrared image of a face, the three images need to be aligned. Three-image alignment means that in face recognition scenarios or payment scenarios using face recognition, the color image, depth image, and infrared image need to be absolutely aligned in time and space, with the same resolution, the same field of view (FOV), and one-to-one pixel alignment.

[0245] For example, aligning the depth map and the color map means adjusting the color map and the depth map to the same resolution so that the pixel granularity of the two images is the same and the pixels on the two images correspond one-to-one.

[0246] During facial recognition payment, the color image, infrared image, and depth image need to be aligned, and the color image and infrared image have a high resolution (commonly 1080x1920), while the depth image has a low resolution (commonly 480x640).

[0247] Related techniques involve downsampling color and infrared images to adapt to the depth map, which results in a loss of FOV and resolution accuracy in the aligned image. However, the method in this embodiment upsamples the depth map to adapt to the color image, thus ensuring FOV and resolution accuracy after alignment.

[0248] Step 302: Based on the adjacency relationship between pixels in the depth texture structure map and the positional correspondence between pixels in the depth texture structure map and pixels in the color texture structure map, obtain the region structure weight of pixels in the depth map.

[0249] Step 303: Upsample the depth map based on the region structure weights to obtain an upsampled depth map.

[0250] Step 304: Perform face recognition based on the color image and the upsampled depth image.

[0251] Using the method provided in this embodiment, the depth map can be upsampled to adapt to the color map and infrared map to obtain a high-resolution, large FOV aligned three-map, thereby improving the speed, accuracy, FOV, and maximum distance of facial recognition payment.

[0252] In summary, the method provided in this embodiment improves the accuracy of face recognition because higher image resolution leads to more accurate facial feature extraction. By using the method provided in this embodiment to obtain three high-resolution, large FOV-aligned images, the accuracy of face recognition can be improved.

[0253] The method provided in this embodiment offers advantages in face recognition. A larger field of view (FOV) allows the computer device to capture an image covering a wider area, resulting in a better user experience. In particular, a larger vertical FOV can accommodate users of a wider range of heights. Using the method provided in this embodiment to obtain high-resolution, large FOV-aligned three images can expand the applicability of face recognition payment.

[0254] The method provided in this embodiment addresses the issue that facial recognition algorithms have minimum pixel requirements for faces. Since faces appear smaller and thus have lower pixel counts at greater distances, a higher resolution improves the clarity of distant faces, thereby increasing the usable distance for the camera to capture facial images during facial recognition payment. Using the method provided in this embodiment to obtain three high-resolution, large-FOV aligned images can extend the usable distance of facial recognition payment devices.

[0255] The following are device embodiments of this application. For details not described in detail in the device embodiments, please refer to the corresponding descriptions in the above method embodiments. They will not be repeated here.

[0256] Figure 9 A schematic diagram of an image processing apparatus provided in an exemplary embodiment of this application is shown. This apparatus can be implemented as all or part of a computer device through software, hardware, or a combination of both. The apparatus includes:

[0257] The first texture module 401 is used to generate a color texture structure map based on a color image and a depth texture structure map based on a depth image, wherein the color image and the depth image are two images of the same scene.

[0258] The first weighting module 402 is used to obtain the region structure weight of the pixels in the depth map based on the adjacency relationship between pixels in the depth texture structure map and the positional correspondence between the pixels in the depth texture structure map and the pixels in the color texture structure map.

[0259] The first upsampling module 403 is used to upsample the depth map based on the region structure weights to obtain an upsampled depth map.

[0260] In an optional embodiment, the first weighting module 402 is used to calculate the sum of the adjacent potential energy and the associated potential energy to obtain a potential energy function. The adjacent potential energy is calculated based on the adjacent weight of the pixels in the depth texture structure map, and the associated potential energy is calculated based on the associated weight of the pixels with the corresponding relationship in the depth texture structure map and the color texture structure map.

[0261] The first weight module 402 is used to adjust the training of the adjacent weights and the associated weights so that the potential function satisfies the training stopping condition;

[0262] The first weighting module 402 is used to determine the adjacency weight when the potential energy function reaches its minimum value as the region structure weight of the pixel in the depth map.

[0263] In one optional embodiment, the depth texture structure map and the color texture structure map each include n pixels at the same position, where n is a positive integer;

[0264] The first weighting module 402 is used to obtain n initial adjacency weights of n pixels in the depth texture structure map; calculate the weighted sum of the Gaussian distribution function of the n pixels in the depth texture structure map and the adjacency weights to obtain the adjacency potential.

[0265] The first weighting module 402 is used to obtain n initial association weights corresponding to the n pixels between the depth texture structure map and the color texture structure map; calculate the weighted sum of the multivariate Gaussian distribution function of the n pixels in the depth texture structure map and the color texture structure map and the association weights to obtain the association potential energy;

[0266] The first weighting module 402 is used to calculate the sum of the adjacent potential energy and the associated potential energy to obtain the potential energy function.

[0267] In one optional embodiment, the training stopping condition includes at least one of: stopping training when the potential function is less than a threshold, or stopping training when the number of training iterations reaches a threshold.

[0268] In an optional embodiment, the first weighting module 402 is used to repeatedly execute the step of obtaining the region structure weight of the pixel in the depth map based on the adjacency relationship between pixels in the depth texture structure map and the position correspondence between the pixels in the depth texture structure map and the pixels in the color texture structure map, so as to obtain at least two region structure weights of the pixel in the depth map.

[0269] The first weighting module 402 is used to calculate the weighted average of the weights of the at least two regional structures to obtain the weighted average of the regional structure weights.

[0270] The first upsampling module 403 is used to upsample the depth map based on the weighted average of the regional structure weights to obtain the upsampled depth map.

[0271] In an optional embodiment, the device further includes:

[0272] The first downsampling module 408 is used to downsample the original color image to obtain the color image, wherein the resolution of the original color image is higher than the resolution of the depth image, and the resolution of the color image is equal to the resolution of the depth image.

[0273] In one alternative embodiment, the depth map includes integer pixels, and the upsampled depth map includes the integer pixels and sub-pixels;

[0274] The first upsampling module 403 is used to calculate the target depth value of the sub-pixel based on the coordinates of the sub-pixel, the depth values ​​of the four integer pixels closest to the sub-pixel, and the region structure weights corresponding to the four integer pixels closest to the sub-pixel.

[0275] The first upsampling module 403 is used to generate the upsampled depth map of the depth image based on the depth value of the integer pixel and the target depth value of the sub-pixel.

[0276] In one alternative embodiment, the region structure weights include three coefficients of a linear equation in two variables;

[0277] The first upsampling module 403 is used to obtain the i-th coordinate of the sub-pixel relative to the i-th integer pixel among the four integer pixels, where i is a positive integer not greater than 4;

[0278] The first upsampling module 403 is used to substitute the i-th coordinate into the i-th region structure weight corresponding to the i-th integer pixel to obtain the i-th region structure weight value.

[0279] The first upsampling module 403 is used to calculate the product of the i-th region structure weight value and the depth value of the i-th integer pixel to obtain the i-th addend;

[0280] The first upsampling module 403 is used to repeat the above steps to obtain the four addends corresponding to the four integer pixels respectively;

[0281] The first upsampling module 403 is used to determine the target depth value of the sub-pixel by summing the four addends.

[0282] Figure 10 A schematic diagram of a face recognition device provided in an exemplary embodiment of this application is shown. This device can be implemented as all or part of a computer device through software, hardware, or a combination of both. The device includes:

[0283] The second texture module 404 is used to generate a color texture structure map based on the color map of the face image to obtain a color texture structure map, and to generate a depth texture structure map based on the depth map of the face image.

[0284] The second weighting module 405 is used to obtain the region structure weight of the pixels in the depth map based on the adjacency relationship between pixels in the depth texture structure map and the positional correspondence between the pixels in the depth texture structure map and the pixels in the color texture structure map.

[0285] The second upsampling module 406 is used to upsample the depth map based on the region structure weights to obtain an upsampled depth map.

[0286] The recognition module 407 is used to perform face recognition based on the color image and the upsampled depth image.

[0287] In an optional embodiment, the second weighting module 405 is used to calculate the sum of the adjacent potential energy and the associated potential energy to obtain a potential energy function. The adjacent potential energy is calculated based on the adjacent weight of the pixels in the depth texture structure map, and the associated potential energy is calculated based on the associated weight of the pixels with the corresponding relationship in the depth texture structure map and the color texture structure map.

[0288] The second weight module 405 is used to adjust the training of the adjacent weights and the associated weights so that the potential function satisfies the training stopping condition.

[0289] The second weighting module 405 is used to determine the adjacency weight when the potential energy function reaches its minimum value as the region structure weight of the pixel in the depth map.

[0290] In one optional embodiment, the depth texture structure map and the color texture structure map each include n pixels at the same position, where n is a positive integer;

[0291] The second weighting module 405 is used to obtain n initial adjacency weights of n pixels in the depth texture structure map; calculate the weighted sum of the Gaussian distribution function of the n pixels in the depth texture structure map and the adjacency weights to obtain the adjacency potential.

[0292] The second weighting module 405 is used to obtain n initial association weights corresponding to the n pixels between the depth texture structure map and the color texture structure map; calculate the weighted sum of the multivariate Gaussian distribution function of the n pixels in the depth texture structure map and the color texture structure map and the association weights to obtain the association potential energy;

[0293] The second weighting module 405 is used to calculate the sum of the adjacent potential energy and the associated potential energy to obtain the potential energy function.

[0294] In one optional embodiment, the training stopping condition includes at least one of: stopping training when the potential function is less than a threshold, or stopping training when the number of training iterations reaches a threshold.

[0295] In an optional embodiment, the second weight module 405 is used to repeatedly execute the step of obtaining the region structure weight of the pixel in the depth map based on the adjacency relationship between pixels in the depth texture structure map and the position correspondence between the pixels in the depth texture structure map and the pixels in the color texture structure map, so as to obtain at least two region structure weights of the pixel in the depth map.

[0296] The second weighting module 405 is used to calculate the weighted average of the weights of the at least two regional structures to obtain the weighted average of the regional structure weights.

[0297] The second upsampling module 406 is used to upsample the depth map based on the weighted average of the regional structure weights to obtain the upsampled depth map.

[0298] In an optional embodiment, the device further includes:

[0299] The second downsampling module 409 is used to downsample the original color image to obtain the color image, wherein the resolution of the original color image is higher than the resolution of the depth image, and the resolution of the color image is equal to the resolution of the depth image.

[0300] In one alternative embodiment, the depth map includes integer pixels, and the upsampled depth map includes the integer pixels and sub-pixels;

[0301] The second upsampling module 406 is used to calculate the target depth value of the sub-pixel based on the coordinates of the sub-pixel, the depth values ​​of the four integer pixels closest to the sub-pixel, and the region structure weights corresponding to the four integer pixels closest to the sub-pixel.

[0302] The second upsampling module 406 is used to generate the upsampled depth map of the depth image based on the depth value of the integer pixel and the target depth value of the sub-pixel.

[0303] In one alternative embodiment, the region structure weights include three coefficients of a linear equation in two variables;

[0304] The second upsampling module 406 is used to obtain the i-th coordinate of the sub-pixel relative to the i-th integer pixel among the four integer pixels, where i is a positive integer not greater than 4;

[0305] The second upsampling module 406 is used to substitute the i-th coordinate into the i-th region structure weight corresponding to the i-th integer pixel to obtain the i-th region structure weight value.

[0306] The second upsampling module 406 is used to calculate the product of the i-th region structure weight value and the depth value of the i-th integer pixel to obtain the i-th addend;

[0307] The second upsampling module 406 is used to repeat the above steps to obtain the four addends corresponding to the four integer pixels respectively;

[0308] The second upsampling module 406 is used to determine the target depth value of the sub-pixel by summing the four addends.

[0309] Figure 11This is a schematic diagram of the structure of a server provided in one embodiment of this application. Specifically, the server 800 includes a Central Processing Unit (CPU) 801, a system memory 804 including Random Access Memory (RAM) 802 and Read-Only Memory (ROM) 803, and a system bus 805 connecting the system memory 804 and the CPU 801. The server 800 also includes a basic input / output system (I / O system) 806 that facilitates the transfer of information between various devices within the computer, and a mass storage device 807 for storing the operating system 813, application programs 814, and other program modules 815.

[0310] The basic input / output system 806 includes a display 808 for displaying information and an input device 809 for inputting user account information, such as a mouse or keyboard. Both the display 808 and the input device 809 are connected to the central processing unit 801 via an input / output controller 810 connected to the system bus 805. The basic input / output system 806 may also include the input / output controller 810 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 810 also provides output to a display screen, printer, or other types of output devices.

[0311] Mass storage device 807 is connected to central processing unit 801 via a mass storage controller (not shown) connected to system bus 805. Mass storage device 807 and its associated computer-readable media provide non-volatile storage for server 800. That is, mass storage device 807 may include computer-readable media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drive.

[0312] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 804 and mass storage device 807 described above can be collectively referred to as memory.

[0313] According to various embodiments of this application, server 800 can also be connected to a remote computer on a network, such as the Internet. That is, server 800 can be connected to network 812 via network interface unit 811 connected to system bus 805, or it can use network interface unit 811 to connect to other types of networks or remote computer systems (not shown).

[0314] This application also provides a terminal, which includes a processor and a memory. The memory stores at least one instruction, which is loaded and executed by the processor to implement the image processing methods provided in the above-described method embodiments. It should be noted that the terminal can be as follows: Figure 12 The provided terminal.

[0315] Figure 12 This illustration shows a structural block diagram of a terminal 900 provided in an exemplary embodiment of this application. The terminal 900 may be a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The terminal 900 may also be referred to as a user account device, portable terminal, laptop terminal, desktop terminal, or other names.

[0316] Typically, terminal 900 includes a processor 901 and a memory 902.

[0317] Processor 901 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 901 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 901 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 901 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 901 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0318] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 902 are used to store at least one instruction, which is executed by the processor 901 to implement the image processing method or image processing method provided in the method embodiments of this application.

[0319] In some embodiments, the terminal 900 may also optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 903 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, a positioning assembly 908, and a power supply 909.

[0320] Peripheral device interface 903 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 901 and memory 902. In some embodiments, processor 901, memory 902 and peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 901, memory 902 and peripheral device interface 903 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0321] The radio frequency (RF) circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 904 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Exemplarily, the RF circuit 904 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user account identity module card, etc. The RF circuit 904 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 904 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0322] Display screen 905 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 905 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 901 for processing. In this case, display screen 905 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 905, which serves as the front panel of terminal 900; in other embodiments, there may be at least two display screens 905, respectively disposed on different surfaces of terminal 900 or in a folded design; in still other embodiments, display screen 905 may be a flexible display screen, disposed on a curved or folded surface of terminal 900. Furthermore, display screen 905 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 905 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0323] Camera assembly 906 is used to acquire images or videos. Exemplarily, camera assembly 906 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, camera assembly 906 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0324] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user account and environment, converting the sound waves into electrical signals that are input to the processor 901 for processing, or input to the radio frequency circuit 904 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the terminal 900. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0325] The positioning component 908 is used to determine the current geographic location of the terminal 900 in order to enable navigation or LBS (Location Based Service). The positioning component 908 can be a positioning component based on the US GPS (Global Positioning System), China's BeiDou system, or Russia's Galileo system.

[0326] Power supply 909 is used to supply power to the various components in terminal 900. Power supply 909 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 909 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0327] In some embodiments, the terminal 900 further includes one or more sensors 910. The one or more sensors 910 include, but are not limited to: an accelerometer 911, a gyroscope 912, a pressure sensor 913, a fingerprint sensor 914, an optical sensor 915, and a proximity sensor 916.

[0328] Accelerometer 911 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by terminal 900. For example, accelerometer 911 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 901 can control display screen 905 to display the user account interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 911. Accelerometer 911 can also be used to collect motion data from games or user accounts.

[0329] The gyroscope sensor 912 can detect the orientation and rotation angle of the terminal 900. The gyroscope sensor 912, in conjunction with the accelerometer sensor 911, can collect 3D motion data from the user account on the terminal 900. Based on the data collected by the gyroscope sensor 912, the processor 901 can perform the following functions: motion sensing (e.g., changing the UI based on the user account's tilting actions), image stabilization during shooting, game control, and inertial navigation.

[0330] The pressure sensor 913 can be installed on the side bezel of the terminal 900 and / or on the lower layer of the display screen 905. When the pressure sensor 913 is installed on the side bezel of the terminal 900, it can detect the user's grip signal on the terminal 900, and the processor 901 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 913. When the pressure sensor 913 is installed on the lower layer of the display screen 905, the processor 901 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 905. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0331] The fingerprint sensor 914 is used to collect the user's fingerprint. The processor 901 identifies the user's account based on the fingerprint collected by the fingerprint sensor 914, or vice versa. When the user's account is identified as a trusted identity, the processor 901 authorizes the user account to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 914 can be located on the front, back, or side of the terminal 900. When the terminal 900 has physical buttons or a manufacturer's logo, the fingerprint sensor 914 can be integrated with the physical buttons or manufacturer's logo.

[0332] An optical sensor 915 is used to collect ambient light intensity. In one embodiment, the processor 901 can control the display brightness of the display screen 905 based on the ambient light intensity collected by the optical sensor 915. Specifically, when the ambient light intensity is high, the display brightness of the display screen 905 is increased; when the ambient light intensity is low, the display brightness of the display screen 905 is decreased. In another embodiment, the processor 901 can also dynamically adjust the shooting parameters of the camera assembly 906 based on the ambient light intensity collected by the optical sensor 915.

[0333] The proximity sensor 916, also known as a distance sensor, is typically located on the front panel of the terminal 900. The proximity sensor 916 is used to detect the distance between the user account and the front of the terminal 900. In one embodiment, when the proximity sensor 916 detects that the distance between the user account and the front of the terminal 900 is gradually decreasing, the processor 901 controls the display screen 905 to switch from a screen-on state to a screen-off state; when the proximity sensor 916 detects that the distance between the user account and the front of the terminal 900 is gradually increasing, the processor 901 controls the display screen 905 to switch from a screen-off state to a screen-on state.

[0334] Those skilled in the art will understand that Figure 12 The structure shown does not constitute a limitation on terminal 900, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0335] The memory further includes one or more programs stored in the memory, and the one or more programs include methods for performing the image processing method and / or face recognition method provided in the embodiments of this application.

[0336] This application also provides a computer device, which includes a processor and a memory. The memory storage medium stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the image processing method and face recognition method provided in the above-described method embodiments.

[0337] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, which is loaded and executed by a processor to implement the image processing method and face recognition method provided in the above-described method embodiments.

[0338] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image processing method and face recognition method provided in the above-described optional implementations.

[0339] It should be understood that "multiple" as mentioned herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. The above are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: A color texture structure map is generated based on a color image, and a depth texture structure map is generated based on a depth image. The color image and the depth image are two images of the same scene. The depth texture structure map and the color texture structure map each include n pixels at the same position, where n is a positive integer. Obtain n initial adjacency weights for the n pixels in the depth texture structure graph; calculate the weighted sum of the Gaussian distribution function of the n pixels in the depth texture structure graph and the adjacency weights to obtain the adjacency potential. Obtain n initial association weights corresponding to the n pixels between the depth texture structure map and the color texture structure map; calculate the weighted sum of the multivariate Gaussian distribution function of the n pixels in the depth texture structure map and the color texture structure map with the association weights to obtain the association potential energy; The potential energy function is obtained by calculating the sum of the adjacent potential energy and the associated potential energy. Adjust the adjacency weights and correlation weights during training so that the potential function satisfies the training stopping condition; The adjacency weight when the potential energy function reaches its minimum value is determined as the region structure weight of the pixel in the depth map. The region structure weight is used to characterize the numerical change relationship of pixel values ​​between each pixel in the depth texture structure map, as well as the numerical change relationship of pixel values ​​between two pixels at the same pixel position in the depth texture structure map and the color texture structure map. An upsampled depth map is obtained by upsampling the depth map based on the region structure weights.

2. The method according to claim 1, characterized in that, The training stopping conditions include at least one of the following: stopping training when the potential function is less than a threshold, or stopping training when the number of training iterations reaches a threshold.

3. The method according to claim 1, characterized in that, The method further includes: Repeat the step of obtaining the region structure weights of pixels in the depth map based on the adjacency relationship between pixels in the depth texture structure map and the position correspondence between pixels in the depth texture structure map and pixels in the color texture structure map, to obtain at least two region structure weights of pixels in the depth map; Calculate the weighted average of the weights of the at least two regional structures to obtain the weighted average of the regional structure weights; The process of upsampling the depth map based on the region structure weights to obtain an upsampled depth map includes: The upsampled depth map is obtained by upsampling the depth map based on the weighted average of the region structure weights.

4. The method according to claim 1, characterized in that, The method further includes: The original color image is downsampled to obtain the color image, wherein the resolution of the original color image is higher than the resolution of the depth image, and the resolution of the color image is equal to the resolution of the depth image.

5. The method according to claim 1, characterized in that, The depth map includes integer pixels, and the upsampled depth map includes both integer pixels and sub-pixels. The process of upsampling the depth map based on the region structure weights to obtain an upsampled depth map includes: The target depth value of the sub-pixel is calculated based on the coordinates of the sub-pixel, the depth values ​​of the four nearest integer pixels to the sub-pixel, and the region structure weights corresponding to the four nearest integer pixels to the sub-pixel. The upsampled depth map of the depth image is generated based on the depth value of the integer pixel and the target depth value of the sub-pixel.

6. The method according to claim 5, characterized in that, The regional structure weights include the three coefficients of a linear equation in two variables; The calculation of the target depth value of the sub-pixel based on the coordinates of the sub-pixel, the depth values ​​of the four nearest integer pixels to the sub-pixel, and the region structure weights corresponding to the four nearest integer pixels to the sub-pixel includes: Obtain the i-th coordinate of the sub-pixel relative to the i-th integer pixel among the four integer pixels, where i is a positive integer not greater than 4; Substitute the i-th coordinate into the i-th region structure weight corresponding to the i-th integer pixel to obtain the i-th region structure weight value; Calculate the product of the i-th region structure weight value and the depth value of the i-th integer pixel to obtain the i-th addend; Repeat the above steps to obtain the four addends corresponding to the four integer pixels; The sum of the four addends is determined as the target depth value of the sub-pixel.

7. A face recognition method, characterized in that, The method includes: A color texture structure map is generated from the color image of the face image, and a depth texture structure map is generated from the depth image of the face image. The depth texture structure map and the color texture structure map each include n pixels at the same position, where n is a positive integer. Obtain n initial adjacency weights for the n pixels in the depth texture structure graph; calculate the weighted sum of the Gaussian distribution function of the n pixels in the depth texture structure graph and the adjacency weights to obtain the adjacency potential. Obtain n initial association weights corresponding to the n pixels between the depth texture structure map and the color texture structure map; calculate the weighted sum of the multivariate Gaussian distribution function of the n pixels in the depth texture structure map and the color texture structure map with the association weights to obtain the association potential energy; The potential energy function is obtained by calculating the sum of the adjacent potential energy and the associated potential energy. Adjust the adjacency weights and correlation weights during training so that the potential function satisfies the training stopping condition; The adjacency weight when the potential energy function reaches its minimum value is determined as the region structure weight of the pixel in the depth map. The region structure weight is used to characterize the numerical change relationship of pixel values ​​between each pixel in the depth texture structure map, as well as the numerical change relationship of pixel values ​​between two pixels at the same pixel position in the depth texture structure map and the color texture structure map. An upsampled depth map is obtained by upsampling the depth map based on the region structure weights. Face recognition is performed based on the color image and the upsampled depth image.

8. An image processing apparatus, characterized in that, The device includes: The first texture module is used to generate a color texture structure map based on a color image and a depth texture structure map based on a depth image. The color image and the depth image are two images of the same scene. The depth texture structure map and the color texture structure map each include n pixels in the same position, where n is a positive integer. The first weighting module is used to obtain n initial adjacency weights for n pixels in the depth texture structure map; calculate the weighted sum of the Gaussian distribution function of the n pixels in the depth texture structure map and the adjacency weights to obtain the adjacency potential; obtain n initial association weights for the n pixels corresponding to the n pixels between the depth texture structure map and the color texture structure map; calculate the weighted sum of the multivariate Gaussian distribution function of the n pixels in the depth texture structure map and the color texture structure map and the association weights to obtain the association potential; calculate the sum of the adjacency potential and the association potential to obtain the potential energy function; adjust and train the adjacency weights and the association weights so that the potential energy function satisfies the training stopping condition; determine the adjacency weight when the potential energy function reaches its minimum value as the region structure weight of the pixel in the depth map, wherein the region structure weight is used to characterize the numerical change relationship of pixel values ​​between each pixel in the depth texture structure map, and the numerical change relationship of pixel values ​​between two pixels at the same pixel position between the depth texture structure map and the color texture structure map; The first upsampling module is used to upsample the depth map based on the region structure weights to obtain an upsampled depth map.

9. A face recognition device, characterized in that, The device includes: The second texture module is used to generate a color texture structure map based on the color map of the face image and to generate a depth texture structure map based on the depth map of the face image. The depth texture structure map and the color texture structure map each include n pixels at the same position, where n is a positive integer. The second weighting module is used to obtain n initial adjacency weights for n pixels in the depth texture structure map; calculate the weighted sum of the Gaussian distribution function of the n pixels in the depth texture structure map and the adjacency weights to obtain the adjacency potential; obtain n initial association weights for the n pixels corresponding to the n pixels between the depth texture structure map and the color texture structure map; calculate the weighted sum of the multivariate Gaussian distribution function of the n pixels in the depth texture structure map and the color texture structure map and the association weights to obtain the association potential; calculate the sum of the adjacency potential and the association potential to obtain the potential energy function; adjust and train the adjacency weights and the association weights so that the potential energy function satisfies the training stopping condition; determine the adjacency weight when the potential energy function reaches its minimum value as the region structure weight of the pixel in the depth map, and the region structure weight is used to characterize the numerical change relationship of pixel values ​​between each pixel in the depth texture structure map, and the numerical change relationship of pixel values ​​between two pixels at the same pixel position between the depth texture structure map and the color texture structure map; The second upsampling module is used to upsample the depth map based on the region structure weights to obtain an upsampled depth map. The recognition module is used to perform face recognition based on the color image and the upsampled depth image.

10. A computer device, characterized in that, The computer device includes a processor and a memory, wherein the memory stores at least one program, which is loaded and executed by the processor to implement the image processing method as described in any one of claims 1 to 6, or the face recognition method as described in claim 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one program, which is loaded and executed by a processor to implement the image processing method as described in any one of claims 1 to 6, or the face recognition method as described in claim 7.

12. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform an image processing method as described in any one of claims 1 to 6, or a face recognition method as described in claim 7.

Images