An image processing method, an image recognition method, an image processing device, and a storage medium

Abstract

The application discloses an image processing method, an image recognition method, a device and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: constructing a tree structure between prototype feature vectors of a plurality of categories and M space feature vectors, wherein each category of the plurality of categories has one or more prototype feature vectors, the prototype feature vectors are represented by leaf nodes in the tree structure, the space feature vectors are represented by non-leaf nodes of the tree structure, and M is a positive integer. The tree structure between the prototype feature vectors of the plurality of categories and the M space feature vectors is used for hierarchical modeling, so that the prototype feature vectors learn the hierarchical relationship between different training images, form a more effective feature space expression, improve the generalization ability of the prototype feature vectors, and are suitable for the determination of categories of a larger number or categories of target images. The accuracy of an image recognition model is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image processing method, an image recognition method, an apparatus, and a storage medium. Background Technology

[0002] With the continuous development of image recognition technology, facial recognition is becoming increasingly popular as a method for identity verification. Due to the rise of social media, more and more users are uploading photos of their lives to these platforms, making facial images and features relatively easy to obtain. Image recognition models used for facial recognition are also vulnerable to attacks that forge faces, such as photo attacks or video attacks.

[0003] To improve the security of image recognition models and successfully resist these face attacks, domain adversarial or metric learning methods are used to align features from different domains to achieve liveness detection on face images used by the image recognition model. However, aligning domain features can lead to feature loss. For example, directly aligning features based on differences in lighting intensity in the shooting environment can result in the loss of brightness or affine features, causing the accuracy of the image recognition model to reach a bottleneck. Summary of the Invention

[0004] This application provides an image processing method, an image recognition method, an apparatus, and a storage medium to overcome the bottleneck of analysis accuracy in image recognition models.

[0005] In view of the above, the first aspect of this application provides an image processing method, comprising:

[0006] Obtain the feature vector of the training image, the prototype feature vectors of multiple categories, and M spatial feature vectors. The prototype feature vectors of multiple categories and the M spatial feature vectors form a tree structure. Each category has one or more prototype feature vectors. The prototype feature vectors are represented by the leaf nodes of the tree structure, and the spatial feature vectors are represented by the non-leaf nodes of the tree structure. M is a positive integer.

[0007] Based on the similarity between the feature vectors of the training images and the prototype feature vectors of multiple categories, update the prototype feature vectors of multiple categories.

[0008] Based on the relationship between leaf nodes and non-leaf nodes, update the prototype feature vectors of multiple categories and M spatial feature vectors;

[0009] The updated prototype feature vectors of multiple categories are used to determine the category of the target image.

[0010] A second aspect of this application provides an image recognition method, comprising:

[0011] Acquire the target image;

[0012] The category of the target image is determined based on the similarity between the target image and the prototype feature vectors of multiple categories, wherein the prototype feature vectors of the multiple categories are obtained according to the image processing method provided in the first aspect above.

[0013] A third aspect of this application provides an image processing apparatus, comprising:

[0014] The acquisition unit is used to acquire the feature vector of the training image, the prototype feature vectors of multiple categories, and M spatial feature vectors. The prototype feature vectors of multiple categories and the M spatial feature vectors form a tree structure. Each category has one or more prototype feature vectors. The prototype feature vectors are represented by the leaf nodes of the tree structure, and the spatial feature vectors are represented by the non-leaf nodes of the tree structure. M is a positive integer.

[0015] The update unit is used to update the prototype feature vectors of multiple categories based on the similarity between the feature vectors of the training image and the prototype feature vectors of multiple categories.

[0016] The update unit is also used to update the prototype feature vectors of multiple categories and M spatial feature vectors based on the relationship between leaf nodes and non-leaf nodes.

[0017] The updated prototype feature vectors of multiple categories are used to determine the category of the target image.

[0018] In one possible implementation of the third aspect, the updating unit is specifically used for:

[0019] Based on the similarity between the feature vector of the training image and the prototype feature vectors of multiple categories, the target prototype feature vector is determined. The target prototype feature vector is the prototype feature vector with the highest similarity to the feature vector of the training image among the prototype feature vectors of multiple categories.

[0020] The target prototype feature vector is updated by increasing the similarity between the feature vector of the training image and the feature vector of the target prototype.

[0021] In one possible implementation of the third aspect, the updating unit is further configured to update the prototype feature vectors of multiple categories by reducing the difference between the first similarity and the second similarity. The first similarity is the similarity between the feature vector of the training image and any one of the prototype feature vectors of the multiple categories. The second similarity is the similarity between the feature vector of the enhanced image and any one of the prototype feature vectors. The enhanced image is obtained by performing data augmentation processing on the training image.

[0022] In one possible implementation of the third aspect, the updating unit is specifically used for:

[0023] The first non-leaf node is determined from the non-leaf nodes. The first non-leaf node represents the nearest common parent node of the leaf node of the first prototype feature vector and the leaf node of the second prototype feature vector. The first prototype feature vector is the K nearest neighbor feature vector of the second prototype feature vector, and the second prototype feature vector is the K nearest neighbor feature vector of the first prototype feature vector. Both the first prototype feature vector and the second prototype feature vector belong to the prototype feature vector, and K is a positive integer.

[0024] By increasing the similarity between the first prototype feature vector and the spatial feature vector represented by the first non-leaf node, the prototype feature vectors of multiple categories and M spatial feature vectors are updated.

[0025] In one possible implementation of the third aspect, the updating unit is further configured to:

[0026] The second non-leaf node is determined from the non-leaf nodes. The second non-leaf node is the nearest common parent node of the leaf node representing the first prototype feature vector, the leaf node representing the second prototype feature vector, and the leaf node representing the third prototype feature vector. The third prototype feature vector does not belong to the K nearest neighbor feature vector of the first prototype feature vector, but belongs to the K nearest neighbor feature vector of the second prototype feature vector.

[0027] By reducing at least one of the following similarities: the similarity between the first prototype feature vector and the spatial feature vector represented by the second non-leaf node, the similarity between the second prototype feature vector and the spatial feature vector represented by the second non-leaf node, and the similarity between the third prototype feature vector and the spatial feature vector represented by the first non-leaf node, the prototype feature vector of multiple categories and M spatial feature vectors are updated.

[0028] In one possible implementation of the third aspect, the updating unit is further configured to update the prototype feature vectors of multiple categories and M spatial feature vectors by reducing the similarity between the prototype feature vectors represented by the leaf nodes and the spatial feature vectors represented by the root node, wherein the root node is one of the non-leaf nodes.

[0029] In one possible implementation of the third aspect, the updating unit is further configured to update the prototype feature vectors of multiple categories and M spatial feature vectors by reducing the similarity between the prototype feature vectors represented by any two leaf nodes.

[0030] In one possible implementation of the third aspect, the updating unit is specifically used for:

[0031] Determine the predicted probability of each non-leaf node, where the predicted probability is the probability that the non-leaf node is the nearest common parent node of the leaf node representing the first prototype feature vector and the leaf node representing the second prototype feature vector.

[0032] Determine the sampling result for each non-leaf node, where the sampling result is the sum of the predicted probability and the random probability;

[0033] The first non-leaf node is determined from the non-leaf nodes based on the sampling results of the non-leaf nodes.

[0034] In one possible implementation of the third aspect, the updating unit is further configured to update the feature vectors of the training image and the feature vectors of the enhanced image by increasing the similarity between the feature vectors of the enhanced image and the feature vectors of the training image.

[0035] In one possible implementation of the third aspect, the acquisition unit is further configured to acquire the true category of the training image;

[0036] The analysis unit is also used to analyze the similarity between the feature vector of the training image and the prototype feature vector of the multiple categories using a classifier, and to determine the predicted category of the training image;

[0037] The update unit is also used to update the classifier based on the true category of the training image and the predicted category of the training image.

[0038] The update unit is also used to determine the predicted category of the training image based on the similarity between the feature vector of the training image and the prototype feature vectors of multiple categories.

[0039] A fourth aspect of this application provides an image recognition device, comprising:

[0040] The acquisition unit is used to acquire the target image;

[0041] The analysis unit is used to determine the category of the target image based on the similarity between the target image and the prototype feature vectors of multiple categories, wherein the prototype feature vectors of the multiple categories are obtained according to the image processing method provided in the first aspect above.

[0042] The fifth aspect of this application provides a computer device, including: a memory, a processor, and a bus system;

[0043] The memory is used to store programs;

[0044] The processor is used to execute programs in memory, and the processor is used to execute the methods mentioned above according to the instructions in the program code;

[0045] Bus systems are used to connect memory and processor to enable communication between them.

[0046] The sixth aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described above.

[0047] A seventh aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the above aspects.

[0048] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0049] In this embodiment, after obtaining the feature vectors of the training image, prototype feature vectors of multiple categories, and M spatial feature vectors, the prototype feature vectors of multiple categories are updated based on the similarity between the feature vectors of the training image and the prototype feature vectors of multiple categories. The prototype feature vectors of multiple categories and the M spatial feature vectors form a tree structure. Each category has one or more prototype feature vectors, represented by leaf nodes in the tree structure, and the spatial feature vectors by non-leaf nodes. M is a positive integer. Based on the relationship between leaf nodes (prototype feature vectors) and non-leaf nodes (spatial feature vectors) in the tree structure, the prototype feature vectors of multiple categories and the M spatial feature vectors are updated. The updated prototype feature vectors of multiple categories are used to determine the category of the target image. By constructing a tree structure between the prototype feature vectors of multiple categories and the M spatial feature vectors, hierarchical modeling is performed, enabling the prototype feature vectors to learn the hierarchical relationships between different training images, forming a more effective feature space representation, and improving the generalization ability of the prototype feature vectors to be applicable to the determination of the category of a larger number or more categories of target images. Furthermore, due to the effective feature space representation of the prototype feature vector, it helps to improve the accuracy of the image recognition model, thereby breaking through the accuracy bottleneck of the image recognition model. Attached Figure Description

[0050] Figure 1 This is a schematic diagram illustrating an application scenario of the image recognition model in this application embodiment;

[0051] Figure 2 This is a schematic diagram of an application architecture for the image processing method in the embodiments of this application;

[0052] Figure 3 This is a schematic diagram of the image processing network to be trained in an embodiment of this application;

[0053] Figure 4 This is a schematic diagram of one embodiment of the image processing method in this application;

[0054] Figure 5aThis is a schematic diagram of a training image to be processed in an embodiment of this application;

[0055] Figure 5b This is a schematic diagram of a training image in an embodiment of this application;

[0056] Figure 6a This is a schematic diagram of the encoding module in an embodiment of this application;

[0057] Figure 6b This is another schematic diagram of the encoding module in an embodiment of this application;

[0058] Figure 7a This is a schematic diagram of a spatial feature vector in an embodiment of this application;

[0059] Figure 7b This is another schematic diagram of spatial feature vectors in the embodiments of this application;

[0060] Figure 8 This is one embodiment of the face liveness detection method in this application;

[0061] Figure 9 This is a schematic diagram of one embodiment of the image processing apparatus in this application;

[0062] Figure 10 This is a schematic diagram of one embodiment of the image recognition device in this application;

[0063] Figure 11 This is a schematic diagram of another embodiment of the image processing device or image recognition device in this application;

[0064] Figure 12 This is a schematic diagram of another embodiment of the image processing device or image recognition device in the present application. Detailed Implementation

[0065] This application provides an image processing method, an image recognition method, an apparatus, and a storage medium to overcome the bottleneck of analysis accuracy in image recognition models.

[0066] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0067] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0068] With the continuous development of image recognition technology, face liveness detection is becoming increasingly popular as an identity verification method. Due to the rise of social media, more and more users upload photos, making facial images and features readily available information. This significantly increases the probability of attacks on facial recognition systems based on facial images and features. To improve the security of facial recognition systems and successfully defend against these attacks, the artificial intelligence (AI) models in these systems employ domain adversarial or metric learning methods to align features from different domains to achieve liveness detection on the facial images used for recognition. However, aligning domain features in the image recognition model can lead to feature loss. For example, directly aligning features introduced by varying lighting intensities in the photographing environment can result in the loss of brightness or affine features, causing a bottleneck in the accuracy of face liveness detection.

[0069] To address this issue, this application proposes that after obtaining the feature vectors of the training image, the prototype feature vectors of multiple categories, and M spatial feature vectors, the prototype feature vectors of multiple categories are updated based on the similarity between the feature vectors of the training image and the prototype feature vectors of multiple categories. The prototype feature vectors of multiple categories and the M spatial feature vectors form a tree structure. Each category has one or more prototype feature vectors. The prototype feature vectors are represented by leaf nodes in the tree structure, and the spatial feature vectors are represented by non-leaf nodes in the tree structure. M is a positive integer. Based on the relationship between the leaf nodes (prototype feature vectors) and non-leaf nodes (spatial feature vectors) in the tree structure, the prototype feature vectors of multiple categories and the M spatial feature vectors are updated. The updated prototype feature vectors of multiple categories are used to determine the category of the target image. By constructing a tree structure between prototype feature vectors of multiple categories and M spatial feature vectors, a more effective feature space representation is formed. By utilizing the tree structure between prototype feature vectors of multiple categories and M spatial feature vectors, the prototype feature vectors of multiple categories and M spatial feature vectors are updated, reducing feature loss during the learning process, improving the accuracy of the image recognition model, and thus breaking through the accuracy bottleneck of the image recognition model.

[0070] The image processing or image recognition methods in the various optional embodiments of this application are implemented based on artificial intelligence (AI) technology. AI is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0071] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0072] Computer vision (CV) is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, three-dimensional (3D) technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), autonomous driving, intelligent transportation, and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0073] This application also relates to cloud technology. Cloud technology refers to a managed technology that unifies hardware, software, network, and other system resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing.

[0074] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology applied to cloud computing business models. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of internet behavior, every item may possess its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing. The cloud technology involved in this application mainly refers to image recognition and other similar processes that may occur between terminal devices or servers via the "cloud."

[0075] For ease of understanding, some terms used in this application are explained below.

[0076] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence (AI) and the fundamental way to endow computers with intelligence; its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning. With the research and advancement of AI technology, it is being researched and applied in multiple fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with technological development, AI will be applied in even more fields and play an increasingly important role.

[0077] Neural Networks: Artificial Neural Networks (ANNs) are composed of numerous neurons connected by adjustable connection weights. They are characterized by large-scale parallel processing, distributed information storage, and good self-organization and self-learning capabilities.

[0078] A convolutional layer (Conv) is a layered structure in a convolutional neural network (CNN) consisting of several convolutional units. A CNN is a type of feedforward neural network that includes at least two neural network layers. Each neural network layer contains several neurons, which are arranged in layers. There are no connections between neurons in the same layer, and information is transmitted between layers in only one direction.

[0079] Backpropagation: Forward propagation refers to the feedforward processing of the model, while backpropagation is the opposite of forward propagation, referring to updating the weight parameters of each layer of the model based on the model's output. For example, if the model includes an input layer, hidden layers, and an output layer, then forward propagation means processing the data in the order of input layer-hidden layer-output layer, while backpropagation means updating the weight parameters of each layer sequentially in the order of output layer-hidden layer-input layer.

[0080] U-Net Network: U-Net is a variant of a fully convolutional network. Therefore, the U-Net network structure is similar to that of the FCN network structure, consisting of a downsampling stage and an upsampling stage. The network structure only contains convolutional layers and pooling layers, without fully connected layers. The shallower, high-resolution layers in the network are used to solve the pixel localization problem, while the deeper layers are used to solve the pixel classification problem, thereby enabling semantic-level segmentation of images.

[0081] DenseNet Network: DenseNet is a classification network that followed ResNet. Its network structure is initially similar to ResNet, starting with a large-scale convolution followed by a pooling layer; then several sub-modules (DenseBlock and TransitionLayer); and finally a pooling layer and a fully connected layer. Taking DenseNet-121 as an example, let's look at its network structure: DenseNet-121 consists of 121 weighted layers, including 4 Denseblocks, totaling 2×(6+12+24+16) = 116 weighted layers, plus 1 convolutional layer for the initial input + 3 transition layers + the final fully connected layer for the output, for a total of 121 layers; during training, the DenseNet-BC structure is used, with a compression factor of 0.5 and a growth rate k = 32; the initial convolutional layer has 2k channels, which reduces the 224×224 input image to 112×112 after 7×7 convolutions; Denseblocks are composed of stacked layers, all with the same size: two conv layers of 1×1 + 3×3 (each conv = BN + ReLU + Conv); transition layers are formed between Denseblocks, which reduce the number of channels using 1×1 convolutional layers and use average pooling layers with a stride of 2 to halve the height and width. Finally, the output is obtained through 1000 softmax passes of global average pooling + fully connected layers.

[0082] ResNet network: Its network structure typically involves a large-scale convolution followed by a pooling layer; then several consecutive sub-modules (DenseBlock and TransitionLayer); finally, a pooling layer and a fully connected layer. Taking ResNet101 as an example, its network structure is explained as follows: ResNet101 has 3+4+23+3=33 building blocks, each block has 3 layers, so there are 33x3=99 layers. Adding the first convolutional layer conv1 and the final fully connected layer (for classification), the total is 99+1+1=101 layers.

[0083] Supervised learning is a training method in machine learning that involves adjusting the parameters of a classifier using a set of samples with known classes to achieve the desired performance. It is also known as supervised training or teacher-guided learning, and is a machine learning task that infers a function from labeled training data. Supervised learning models are generally classified into two types: First, by model form: probabilistic models and non-probabilistic models; second, by whether they model the distribution of observed variables: discriminative models and generative models.

[0084] Semi-supervised learning: A training method in machine learning. Its basic idea is to train on labeled data, then predict unlabeled data, take the sample with the highest prediction confidence and directly label it, and then include this type of sample in the current training sample to continue training until the model's prediction results no longer change.

[0085] Given that this application involves some technical terms, these terms will be introduced below.

[0086] Liveness detection: A method for determining the true physiological characteristics of an individual in identity verification scenarios. In facial recognition applications, liveness detection verifies whether a user is a real, living person by detecting combinations of actions such as blinking, opening their mouth, shaking their head, and nodding, using facial landmark localization and face detection technologies. It effectively defends against common attacks using photos, videos, face swapping, masks, occlusions, 3D animations, and screen captures, thus helping users identify fraudulent activities and protecting their interests.

[0087] Facial attacks: These can deceive facial recognition systems by presenting the camera with a photo, video, or 3D mask of the target person. Makeup or plastic surgery can also be used. However, due to the high exposure and low cost of high-resolution digital cameras, using photos and videos is the most common type of attack.

[0088] Photo attacks: Photo attacks involve displaying a photo of the person being attacked on the sensors of a facial recognition system.

[0089] Video attack: An attacker can play a legitimate user's video on any device that copies the video and then present it to sensors / cameras.

[0090] 3D masking attack: In this type of attack, the attacker constructs a 3D reconstruction of a face and presents it to the sensor / camera.

[0091] Other attacks: makeup, cosmetic surgery, and other attacks that mimic the victim.

[0092] The method provided in this application is applicable to, for example... Figure 1 The communication system shown.

[0093] Figure 1 This diagram illustrates an application scenario of the image recognition model provided in this application, which includes a front-end device A, a back-end device, and a front-end device B. The front-end device A acquires a target image and uploads it to the back-end device. The back-end device runs the image recognition model proposed in this application to obtain the category of the target image, and then sends the category of the target image to the front-end device B.

[0094] It is understandable that front-end device A, back-end device, and front-end device B can be the same device or different devices; there is no restriction here.

[0095] See Figure 2 , Figure 2This is a schematic diagram of an optional architecture for an image processing solution provided in this application embodiment. To support an image processing solution, terminal device 100 connects to server 300 via network 200, and server 300 connects to database 400. Network 200 can be a wide area network (WAN), a local area network (LAN), or a combination of both. The client for implementing the image processing solution is deployed on terminal device 100. The client can run on terminal device 100 via a browser or as a standalone application (APP). The specific form of the client is not limited here. Server 300 involved in this application can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal device 100 can be a smartphone, tablet, laptop, PDA, personal computer, smart TV, smartwatch, in-vehicle device, wearable device, etc., but is not limited to these. Terminal device 100 and server 300 can be directly or indirectly connected via network 200 through wired or wireless communication, which is not limited in this application. The number of server 300 and terminal device 100 is also not limited. The solution provided in this application can be completed independently by terminal device 100, independently by server 300, or jointly by terminal device 100 and server 300, which is not specifically limited in this application. In short, database 400 can be regarded as an electronic filing cabinet—a place to store electronic files, where users can perform operations such as adding, querying, updating, and deleting data in the files. The so-called "database" is a collection of data stored together in a certain way, which can be shared by multiple users, has the lowest possible redundancy, and is independent of applications. Database management system (DBMS) is a computer software system designed for managing databases, and generally has basic functions such as storage, retrieval, security, and backup.Database management systems can be categorized based on the database model they support, such as relational or Extensible Markup Language (XML); or based on the type of computer they support, such as server clusters or mobile phones; or based on the query language used, such as Structured Query Language (SQL) or XQuery; or based on performance priorities, such as maximum scale or highest operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, simultaneously supporting multiple query languages. In this application, database 400 can be used to store feature vectors of training images, prototype feature vectors of multiple categories, and M spatial feature vectors or target images. Of course, the storage location of the feature vectors of training images, prototype feature vectors of multiple categories, and M spatial feature vectors or target images is not limited to a database; for example, it can also be stored in terminal device 100, a blockchain, or a distributed file system of server 300, etc.

[0096] In some embodiments, both the server 300 and the terminal device 100 can execute the image processing method and image recognition method provided in the embodiments of this application.

[0097] In this embodiment, the specific process can be as follows: The terminal device 10 acquires the feature vector of the training image, the prototype feature vectors of multiple categories, and M spatial feature vectors; then the terminal device 100 stores the feature vector of the training image, the prototype feature vectors of multiple categories, and the M spatial feature vectors in the database 400 or in the memory of the terminal device 100; the server 300 acquires the feature vector of the training image, the prototype feature vectors of multiple categories, and the M spatial feature vectors from the database 400 or the terminal device 100; then, based on the similarity between the feature vector of the training image and the prototype feature vectors of multiple categories, the prototype feature vectors of multiple categories are updated; based on the relationship between leaf nodes and non-leaf nodes, the prototype feature vectors of multiple categories and the M spatial feature vectors are updated; wherein, the updated prototype feature vectors of multiple categories are used to determine the category of the target image. Finally, the server 300 can deploy the image recognition model to the terminal device 100 so that the terminal device 100 can call the image recognition model to classify the target image; or, the server 300 can deploy the image recognition model on the image recognition server so that the image recognition server can call the image recognition model to classify the target image.

[0098] In another embodiment, the terminal device 100 independently executes the image processing method provided in this application embodiment. In this embodiment, the specific process can be as follows: The terminal device 100 acquires the feature vector of the training image, prototype feature vectors of multiple categories, and M spatial feature vectors; then, the terminal device 100 updates the prototype feature vectors of multiple categories based on the similarity between the feature vector of the training image and the prototype feature vectors of multiple categories; and updates the prototype feature vectors of multiple categories and the M spatial feature vectors based on the relationship between leaf nodes and non-leaf nodes; wherein the updated prototype feature vectors of multiple categories are used to determine the category of the target image. Finally, the terminal device 100 can deploy the image recognition model to itself, so that the terminal device 100 can call the image recognition model to classify the target image; or, the terminal device 100 can deploy the image recognition model on an image recognition server, so that the image recognition server calls the image recognition model to classify the target image.

[0099] It is understood that in the specific embodiments of this application, the training images or target images involved are face images and / or human image-related data. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0100] The solution provided in this application is applied to image processing. To facilitate an intuitive understanding of the structure of the image recognition model proposed in this application, the following section combines... Figure 3 An introduction to image recognition models:

[0101] Image recognition models include an encoding module, a feature learning module, and a classification module.

[0102] The encoding module is used to encode the input image to obtain the feature vector of the training image, and then input the feature vector of the training image into the feature learning module.

[0103] The feature learning module contains multiple categories of prototype feature vectors and M spatial feature vectors. After processing the feature vectors of the training image using these prototype feature vectors and M spatial feature vectors, the module inputs the processed vectors into the classification model. The prototype feature vectors and M spatial feature vectors form a tree structure. Each category has one or more prototype feature vectors, represented by the leaf nodes of the tree structure, while the spatial feature vectors are represented by the non-leaf nodes.

[0104] The classification module is used to determine the predicted label of the training image by utilizing the similarity between the feature vector of the training image and the prototype feature vector of multiple categories, and to update the classifier based on the predicted label and the true label of the training image.

[0105] The image processing methods mentioned in this application are used to train the feature learning module and classification module in an image recognition model. In some possible scenarios, they can also be used to train the encoding module of an image recognition model, and there are no restrictions here.

[0106] The foregoing Figure 3 In the image recognition model mentioned, the feature learning module can be configured with three different learning objectives, for example:

[0107] Learning objective one is to perform feature enhancement on training images to obtain enhanced images. During the enhancement process, the similarity between the enhanced and training images is ensured by narrowing the distance between their feature vectors. Furthermore, the distribution of the enhanced image's feature vectors in the vector space is ensured to be similar to that of the training image's feature vectors by reducing the difference between the first and second similarities. Specifically, the first similarity is the similarity between the training image's feature vectors and any one of the prototype feature vectors from multiple categories, and the second similarity is the similarity between the enhanced image's feature vectors and the prototype feature vectors from multiple categories.

[0108] Learning objective two aims to improve the similarity between the feature vectors of the training image and the target prototype feature vectors, thereby narrowing the distance between them. The target prototype feature vector is the prototype feature vector among multiple categories that is most similar to the feature vectors of the training image.

[0109] Learning objective three is used to update the prototype feature vectors of multiple categories and M spatial feature vectors, thereby updating the tree structure.

[0110] The image processing and image recognition methods provided in this application can be applied to the training of quality inspection models for industrial parts, which are used to distinguish whether industrial parts have defects. In addition, the image processing and image recognition methods provided in this application can also be applied to the training of image recognition models, which are used to distinguish whether a face in an image is a live face, without any limitation.

[0111] Based on the above introduction, the training method of the image recognition model in this application will be described below, taking the scheme provided in this application for training an image recognition model as an example. When this application is applied to a face liveness detection scenario, the multiple categories in this application include both live and non-live categories. Therefore, the following explanation will focus on the two categories of live and non-live.

[0112] Please see Figure 4 One embodiment of the image recognition training method in this application includes:

[0113] S101. Obtain the feature vector of the training image, the prototype feature vectors of multiple categories, and M spatial feature vectors;

[0114] The feature vector of the training image can be the feature vector obtained by feature acquisition of the training image. The tree structure consists of multiple class prototype feature vectors and M spatial feature vectors. Each class has one or more prototype feature vectors. The prototype feature vectors are represented by the leaf nodes of the tree structure, and the spatial feature vectors are represented by the non-leaf nodes of the tree structure. M is a positive integer.

[0115] Specifically, based on the foregoing Figure 3 The image recognition model shown can obtain the feature vector of the training image in the encoding module of the image recognition model. When the encoding module obtains the feature vector of the training image, the training image can be a preprocessed image.

[0116] When the training images can be preprocessed images, the encoding module specifically implements the following steps:

[0117] First, the training images to be processed are acquired. After obtaining such images... Figure 5a After showing the training image to be processed, determine the region where the face is located in the training image to be processed, such as... Figure 5a The area shown is region a (white outline). Using region a as the center, expand region a by a factor of 1.7 to obtain the following... Figure 5a The training image to be processed is cropped along the frame of region b (black border). The cropped result is then reduced to a size of 255*255, as shown below. Figure 5b The training images shown.

[0118] It is understood that the description of the preprocessing method here is only an example. In actual applications, it should be set according to the specific application scenario. No restrictions are imposed here.

[0119] Then, the training images are encoded to obtain the feature vectors of the training images.

[0120] Since the image recognition model provided in this application can perform correlation analysis based on Euclidean space or hyperbolic space, the feature vector of the training image can be an Euclidean space vector or a hyperbolic space vector, without any restriction here.

[0121] When the image recognition model is based on Euclidean space analysis, the training images are encoded as follows: Figure 6a As shown, feature extraction is performed on the training image to obtain the feature vector of the training image. The feature vector of the training image is a Euclidean space vector.

[0122] When the image recognition model is based on hyperbolic space analysis, the training images are encoded as follows: Figure 6b As shown, feature extraction is performed on the training image to obtain the feature vector of the training image in Euclidean space. Then, spatial transformation is performed on the feature vector of the training image in Euclidean space to obtain the feature vector of the training image in hyperbolic space.

[0123] It is understood that the description of the preprocessing module here is only an example. In actual applications, the specific structure and execution content of the encoding module should be set according to the specific application scenario, and no restrictions are imposed here.

[0124] Specifically, based on the foregoing Figure 3 The image recognition model shown can also obtain prototype feature vectors of multiple categories and M spatial feature vectors in the feature encoding module. When obtaining prototype feature vectors of multiple categories and M spatial feature vectors, the prototype feature vectors of multiple categories and M spatial feature vectors are obtained by analyzing the feature vectors of the training image.

[0125] In addition, it can obtain prototype feature vectors of multiple categories and M spatial feature vectors, and can also directly obtain prototype feature vectors of multiple categories and M spatial feature vectors preset.

[0126] Among them, at least two preset prototype feature vectors to be trained and M spatial feature vectors to be trained are a set of feature vectors whose number of prototype feature vectors and spatial feature vectors are randomly set.

[0127] Among them, the prototype feature vectors of multiple categories and M spatial feature vectors form a set of feature vectors in the target space, and the prototype feature vectors are represented as follows: Where i∈{0,1}, it represents whether the prototype feature vector is a feature vector of a living human image, which can also be understood as the label of the prototype feature vector, and j={0,1,…,K i∠-1} represents different prototype feature vectors. The prototype feature vectors for multiple categories include |P| pre-defined prototype feature vectors to be trained, where |P| = K = K 0 +K 1 .

[0128] M spatial feature vectors can be represented as P none-leaf ={p k |k∈{0,1,…,K′-1}}. Where M is a positive integer, M spatial feature vectors and multiple class prototype feature vectors form a tree structure, the multiple class prototype feature vectors are the leaf nodes of the tree structure, and the M spatial feature vectors are the non-leaf nodes of the tree structure.

[0129] Specifically, in facial recognition detection, the system can be deceived by presenting the camera with a photo, video, or 3D mask of the target person. This can also be achieved through makeup or plastic surgery. These photos, videos, or 3D masks used to deceive the facial recognition system are all considered facial attacks.

[0130] For example, to facilitate understanding, please refer to Figure 7a The following sections introduce the M spatial feature vectors and the prototype feature vectors of multiple categories. The M spatial feature vectors can represent the types of face attacks that have appeared in the training images. For example, the M spatial feature vectors can indicate whether there is face occlusion in the training images (full face occlusion / partial occlusion), or the material used in the face attacks in the training images (paper / silicone / electronic / plastic), or the type of face attack in the training images (2D attack / 3D attack), etc. There are no restrictions here.

[0131] Correspondingly, the prototype feature vectors of multiple categories include Figure 7a The feature vectors of the multiple images shown are shown.

[0132] against Figure 7a The M spatial feature vectors introduced in the text are used to set up a tree structure to obtain, as shown in the figure. Figure 7b The tree structure shown first categorizes the types of face attacks in the training images, then categorizes the materials used in the face attacks, and finally categorizes whether facial occlusion exists in the training images, obtaining a set of prototype feature vectors for multiple categories. Figure 7a and Figure 7b The face images shown refer to the images corresponding to the prototype feature vectors.

[0133] S102. Based on the similarity between the feature vector of the training image and the prototype feature vector of multiple categories, update the prototype feature vector of multiple categories and the feature vector of the training image.

[0134] In this context, the feature vectors of the training image and the prototype feature vectors of multiple categories are both vector representations in the target space. In order to enable the prototype feature vectors to better represent the feature vectors of the training image, the prototype feature vectors of multiple categories and the feature vectors of the training image are updated through contrastive learning.

[0135] Step S102 is described above. Figure 3 The feature learning module shown is executed here. The following will describe learning objective one and learning objective two of the three different learning objectives in conjunction with specific implementation methods:

[0136] Learning Objective 1:

[0137] Optionally, while constraining the distance between the feature vector of the training image and at least two preset prototype feature vectors to be trained, an augmented image of the training image can be introduced, and the target prototype feature vector can be updated using the feature vector of the augmented image and the feature vector of the training image.

[0138] For example, the training image is first enhanced with features to obtain the enhanced image.

[0139] Feature enhancement can include randomly cropping the image corresponding to the training feature vector, adjusting the brightness and contrast of the image corresponding to the training feature vector, flipping the image corresponding to the training feature vector, or performing an affine transformation on the image corresponding to the training feature vector, etc., without any restrictions.

[0140] The feature learning module to be trained can enhance the training feature vector by randomly selecting feature enhancement methods from the feature enhancement list, or by sequentially selecting feature enhancement methods from the feature enhancement list, or by calling code to select feature enhancement methods for the training feature vector according to certain rules. There are no restrictions here.

[0141] It is understood that the description of feature enhancement methods here is only an example. In practical applications, feature enhancement should be performed on the training feature vectors in combination with the specific application scenario. No restrictions are imposed here.

[0142] Learning objective one can ensure the similarity between the augmented image and the training image from two different dimensions.

[0143] Dimension 1: Training images and augmented images should be similar;

[0144] The enhanced image obtained after feature enhancement of the training image should be similar to the training image, and the feature vector of the enhanced image should be similar to the feature vector of the training image.

[0145] Therefore, the loss function between updating the feature vector of the enhanced image and the feature vector of the training image can be expressed as:

[0146] L II-feat =||detach(z)-z aug ||2;

[0147] Where detach(z) represents the feature vector of the training image, z aug This represents the feature vector of the enhanced image.

[0148] Based on this, since the enhanced image is obtained by feature enhancement based on the training image, the similarity between the feature vector of the enhanced image and the feature vector of the target prototype should be consistent with the similarity between the feature vector of the training image and the feature vector of the target prototype.

[0149] In this embodiment, by limiting the similarity between the feature vectors of the training image and the feature vectors of the enhanced image, strong support is provided for obtaining the enhanced image by performing feature enhancement on the training image while retaining the features of the training image, thus avoiding the loss of key features of the training image during the feature enhancement process.

[0150] Dimension 2: The distribution of feature vectors in the enhanced image space should be consistent with the distribution of feature vectors in the vector space of the training image.

[0151] The enhanced image obtained after feature enhancement of the training image should be similar to the training image. The similarity between the feature vector of the enhanced image and any prototype feature vector of multiple categories should be consistent with the similarity between the feature vector of the training image and any prototype feature vector of multiple categories.

[0152] In other words, when the distance between the feature vector of the training image X and the prototype feature vector X is A, the distance between the feature vector of the augmented image Y and the prototype feature vector X should be close to A.

[0153] Specifically, the similarity between the feature vector of the training image and any one of the prototype feature vectors of multiple categories is called the first similarity, and the similarity between the feature vector of the augmented image and any one of the prototype feature vectors of multiple categories is called the second similarity. The update of the feature vectors of the augmented image, the training image, and the target prototype feature vector is performed by reducing the difference between the first and second similarities.

[0154] Its loss function can be expressed as:

[0155] LII-dist =||d H (detach(z),Ρ leaf )-d H (z aug ,Ρ leaf )||2;

[0156] Where detach(z) represents the feature vector of the training image, z aug P represents the feature vector of the enhanced image. leaf d represents the target prototype feature vector. H (detach(z),Ρ leaf ) represents the distance between the feature vector of the training image and any one of the prototype feature vectors of multiple categories, d H (z aug ,Ρ leaf ) represents the distance between the feature vector of the enhanced image and any one of the prototype feature vectors of multiple categories.

[0157] It is understood that the descriptions of prototype feature vectors for multiple categories and feature vectors for training images here are merely examples. In specific application scenarios, the settings should be combined with actual needs, and no restrictions are imposed here.

[0158] In this embodiment of the application, in order to ensure that the enhanced image obtained by feature enhancement of the training image is similar to the training image, the prototype feature vectors of multiple categories are updated by reducing the difference between the first similarity and the second similarity. This makes the similarity between the feature vectors corresponding to the enhanced image and the training image and any prototype feature vector of the multiple categories consistent, thus avoiding the situation where the distribution of the obtained enhanced image and the training image in the vector space is inconsistent.

[0159] Learning Objective Two:

[0160] Specifically, in order to enable the prototype feature vectors to more effectively represent the feature space composed of at least two preset prototype feature vectors to be trained, the distance between the feature vectors of the training image and the preset prototype feature vectors to be trained is constrained by the contrastive learning method.

[0161] For example, taking S prototype feature vectors of multiple categories as an example, this section explains the distance constraint between the feature vector of the training image and at least two preset prototype feature vectors to be trained, where S is an integer greater than 1.

[0162] Feature vector 1 of the training image is any one of the feature vectors in the training image. The similarity between feature vector 1 of the training image and S prototype feature vectors is calculated to obtain S first similarities. The prototype feature vector 1 with the highest similarity among these S first similarities is selected as the target prototype feature vector. The similarity between feature vector 1 of the training image and prototype feature vector 1 is improved to enhance the similarity between feature vector 1 of the training image and the preset prototype feature vector 1 to be trained, thereby updating both feature vector 1 of the training image and prototype feature vector 1.

[0163] Furthermore, this application uses the distance between two feature vectors to describe the similarity between them, that is, using d H (x,y) represents the distance between feature vectors x and y, which is the similarity between feature vectors x and y. No restrictions are imposed here.

[0164] The loss function for updating the feature vectors and prototype feature vectors of the training image can be expressed as:

[0165]

[0166] Among them, z n p represents the feature vector of the training image. ck Among S prototype feature vectors, the one with z n The pre-defined prototype feature vector to be trained with the smallest distance, which is also the target reproducible feature vector; p ij Let i be the j-th prototype feature vector with label i.

[0167] In this embodiment, the similarity between the feature vector of the training image and the prototype feature vector of multiple categories is used to determine the target prototype feature vector that is most similar to the feature vector of the training image. Then, the target prototype feature vector is updated by increasing the similarity between the feature vector of the training image and the target prototype feature vector. This can effectively update the target prototype feature vector based on the current training feature vector, so that the updated target prototype feature vector can more accurately classify the feature vector of the training image.

[0168] Optionally, while constraining the distance between the feature vector of the training image and at least two preset prototype feature vectors to be trained, an augmented image of the training image can be introduced, and the target prototype feature vector can be updated using the feature vector of the augmented image and the feature vector of the training image.

[0169] For example, the training image is first enhanced with features to obtain the enhanced image.

[0170] Feature enhancement can include randomly cropping the image corresponding to the training feature vector, adjusting the brightness and contrast of the image corresponding to the training feature vector, flipping the image corresponding to the training feature vector, or performing an affine transformation on the image corresponding to the training feature vector, etc., without any restrictions.

[0171] The feature learning module to be trained can enhance the training feature vector by randomly selecting feature enhancement methods from the feature enhancement list, or by sequentially selecting feature enhancement methods from the feature enhancement list, or by calling code to select feature enhancement methods for the training feature vector according to certain rules. There are no restrictions here.

[0172] It is understood that the description of feature enhancement methods here is only an example. In practical applications, feature enhancement should be performed on the training feature vectors in combination with the specific application scenario. No restrictions are imposed here.

[0173] Since the enhanced image obtained after feature augmentation of the training image should be similar to the training image, the loss function for updating the feature vector of the enhanced image and the feature vector of the training image can be expressed as:

[0174] L II-feat =||detach(z)-z aug ||2;

[0175] Where detach(z) represents the feature vector of the training image, z aug This represents the feature vector of the enhanced image.

[0176] Based on this, since the enhanced image is obtained by feature enhancement based on the training image, the similarity between the feature vector of the enhanced image and the feature vector of the target prototype should be consistent with the similarity between the feature vector of the training image and the feature vector of the target prototype.

[0177] In this embodiment, by increasing the similarity between the feature vector of the training image and the feature vector of the enhanced image, it provides strong support for obtaining the enhanced image by enhancing the features of the training image while retaining the features of the training image, and avoids the situation of losing the key features of the training image during the feature enhancement process.

[0178] The similarity between the feature vectors of the training image and the target prototype is called the first similarity, and the similarity between the feature vectors of the augmented image and the target prototype is called the second similarity. The feature vectors of the augmented image, the training image, and the target prototype are updated in a way that reduces the difference between the first and second similarities.

[0179] Its loss function can be expressed as:

[0180] L II-dist =||d H (detach(z),Ρ leaf )-d H (z aug ,Ρ leaf )||2;

[0181] Where detach(z) represents the feature vector of the training image, z aug P represents the feature vector of the enhanced image. leaf d represents the target prototype feature vector. H (detach(z),Ρ leaf ) represents the distance between the feature vector of the training image and the feature vector of the target prototype, d H (z aug ,Ρ leaf ) represents the distance between the feature vector of the enhanced image and the feature vector of the target prototype.

[0182] It is understood that the descriptions of prototype feature vectors for multiple categories and feature vectors for training images here are merely examples. In specific application scenarios, the settings should be combined with actual needs, and no restrictions are imposed here.

[0183] In this embodiment of the application, in order to ensure that the enhanced image obtained by feature enhancement of the training image is similar to the training image, the target prototype feature vector is updated by reducing the difference between the first similarity and the second similarity, so that the similarity between the feature vectors corresponding to the enhanced image and the training image and the target prototype feature vector is consistent, thus avoiding the situation where the obtained enhanced image and the training image are similar to different prototype feature vectors.

[0184] S103. Based on the relationship between leaf nodes and non-leaf nodes, update the prototype feature vectors of multiple categories and M spatial feature vectors.

[0185] In this context, leaf nodes are prototype feature vectors, and non-leaf nodes are spatial feature vectors.

[0186] Step S103 is also mentioned above. Figure 3The feature learning module shown is executed here. The following will describe learning objective three, one of the three different learning objectives, in conjunction with a specific implementation method:

[0187] Since the M spatial feature vectors are used to help model the prototype feature vectors of multiple categories hierarchically, and the prototype feature vectors of multiple categories and the M spatial feature vectors form a tree structure, this application proposes to construct triples to constrain the distance between features and the lowest common parent node of samples, thereby optimizing the distance between the prototype feature vectors of multiple categories and the M spatial feature vectors.

[0188] Specifically, firstly, the lowest common parent node of the first prototype feature vector and the second prototype feature vector is selected from the M spatial feature vectors (non-leaf nodes) as the first spatial feature vector (first non-leaf node), where the first prototype feature vector and the second prototype feature vector are directly opposite each other.

[0189] For example, a triplet is constructed based on M spatial feature vectors and prototype feature vectors of multiple categories. For prototype feature vectors of multiple categories, assume there are N prototype feature vectors, where any two prototype feature vectors p i and p j p i For p j The K nearest neighbor samples, and p j Also for p i If the K nearest neighbor sample is p, then p is considered to be p. i and p j If it is correct, then p i and p j For a negative pair, it is denoted as:

[0190]

[0191] Where pi is the i-th prototype feature vector, p j Let N be the j-th prototype feature vector. k (p j ) is p j The K nearest neighbor feature vectors, N k (p i ) is p i The K-nearest neighbor feature vectors.

[0192] In sampling the first spatial feature vector, the non-leaf node corresponding to the first spatial feature vector in the tree structure is the first prototype feature vector (p). i ) and the second prototype feature vector (p jWhen the nearest common parent node of the corresponding leaf node in the tree structure is considered, the first spatial feature vector is considered to be the common parent node of the first prototype feature vector and the second prototype feature vector. The probability of sampling to obtain the common parent node of the first prototype feature vector and the second prototype feature vector can be expressed as:

[0193]

[0194] Where ρ represents the first spatial eigenvector, d H (p i ,ρ) represents the first prototype eigenvector p i The distance d from the first spatial eigenvector ρ H (p j ,ρ) represents the second prototype eigenvector p j The distance between the first spatial eigenvector ρ and the first spatial eigenvector.

[0195] The actual sampling results can be expressed as:

[0196]

[0197] Where, ρ ij Let p represent the spatial feature vector to be trained. i and p j The lowest common parent node (preset prototype feature vector), g ij It is random noise (random probability).

[0198] During the sampling process, certain spatial feature vectors may be sampled repeatedly. To avoid overfitting, random noise is introduced as a perturbation during the sampling process to adjust the sampling probability and prevent overfitting.

[0199] In this embodiment, a predicted probability for each non-leaf node is introduced. Sampling is performed from the prototype feature vectors corresponding to multiple non-leaf nodes. The probability that the sampled leaf node is the first leaf node is the predicted probability. To avoid the risk of overfitting from multiple samplings, which could result in only a fixed number of non-leaf nodes being sampled, a random probability is introduced as a perturbation. The sum of the predicted probability and the random probability of the non-leaf node is used to represent whether the sampled non-leaf node is the first leaf node, thus reducing the probability of overfitting during sampling. This avoids the situation where only a few fixed non-leaf nodes are sampled during the sampling of the first non-leaf node, preventing a comprehensive update of the M spatial feature vectors and multiple prototype feature vectors.

[0200] Based on the collected first spatial feature vector, and assuming the first prototype feature vector and the second prototype feature vector are aligned, the first spatial feature vector, the first prototype feature vector, and the second prototype feature vector are updated by increasing the similarity between the first prototype feature vector and the first spatial feature vector.

[0201] Furthermore, a second spatial feature vector (the feature vector corresponding to the second non-leaf node) can be collected from M spatial feature vectors. The non-leaf node corresponding to the second spatial feature vector in the tree structure is the nearest common parent node of the leaf nodes corresponding to the first prototype feature vector, the second prototype feature vector, and the third prototype feature vector in the tree structure. The third prototype feature vector does not belong to the K nearest neighbor feature vector of the first prototype feature vector, and the third prototype feature vector belongs to the K nearest neighbor feature vector of the second prototype feature vector.

[0202] At this point, the prototype feature vectors of multiple categories and the M spatial feature vectors can be updated by reducing at least one of the following: reducing the similarity between the first prototype feature vector and the second spatial feature vector, reducing the similarity between the second prototype feature vector and the second spatial feature vector, or reducing the similarity between the third prototype feature vector and the first spatial feature vector.

[0203] It is understood that the description of the relationship between the third prototype feature vector and the second and first prototype feature vectors here is only an example. In practical applications, the third prototype feature vector and the second prototype feature vector, as well as the third prototype feature vector and the first prototype feature vector, should all be negative pairs. No restrictions are imposed here.

[0204] Therefore, when updating the prototype feature vectors of multiple categories and M spatial feature vectors based on the relationship between leaf nodes and non-leaf nodes, the loss function can be expressed as:

[0205]

[0206] Where pi is the first prototype feature vector, p j Let pi and p be the second prototype feature vectors. j ρ are mutually opposite pre-defined prototype feature vectors. ij For p i and p j The lowest common parent node (first space eigenvector), p k ρ is the third prototype feature vector. ijk For p k p i and p j The lowest common parent node (second spatial feature vector) is δ, which is the optimized boundary range used to limit the distance between nodes.

[0207] Based on the aforementioned updating of multiple categories of prototype feature vectors and M spatial feature vectors, when the scheme provided in this application is implemented in hyperbolic space, based on the spatial features of hyperbolic space, the leaf nodes (prototype feature vectors) need to be far away from the root node; in order to make the preset prototype feature vectors to be trained as fully as possible represent the possible distribution of training feature vectors, and also to make the leaf nodes (prototype feature vectors) exhibit as large a difference as possible.

[0208] In this embodiment, when the first non-leaf node is the nearest common parent node of the leaf nodes corresponding to the first prototype feature vector and the second prototype feature vector, and the first prototype feature vector and the second prototype feature vector are K nearest neighbor feature vectors, the similarity between the first prototype feature vector and the first non-leaf node (first spatial feature vector) is improved, providing a direction for reasonably and effectively updating the M spatial feature vectors.

[0209] Specifically, the prototype feature vectors of multiple categories and M spatial feature vectors can be updated by reducing the similarity between the prototype feature vectors identified by the leaf nodes and the spatial feature vectors represented by the root nodes, where the root node is a non-leaf node.

[0210] In this embodiment, when the second non-leaf node is the nearest common parent node of the leaf nodes corresponding to the first, second, and third prototype feature vectors, the first and second prototype feature vectors are each other's K nearest neighbor feature vectors. When the third prototype feature vector and the second prototype feature vector are not each other's K nearest neighbor feature vectors, the similarity between the first and second prototype feature vectors and the second leaf node is reduced, or the similarity between the third prototype feature vector and the first leaf node is reduced. This is to make the prototype feature vectors represented by the leaf nodes far away from the spatial feature vectors represented by the non-leaf nodes, so that among the representations of multiple categories of prototype feature vectors and M spatial feature vectors, the prototype feature vectors corresponding to the parent nodes of the leaf nodes can have richer expressions.

[0211] The loss function for updating the prototype feature vectors of multiple categories and the M spatial feature vectors can also be expressed as:

[0212]

[0213] Where, d H (p i ,0) represents the distance between a leaf node and the root node.

[0214] In this embodiment of the application, it is also proposed that by widening the distance between the prototype feature vector represented by the leaf node and the spatial feature vector represented by the root node, the similarity between the prototype feature vector represented by the leaf node and the spatial feature vector represented by the root node can be reduced, thereby enabling M spatial feature vectors to have a richer expression when they have the same number of levels.

[0215] Specifically, the prototype feature vectors of multiple categories and M spatial feature vectors can be updated by reducing the similarity between the prototype feature vectors identified by any two leaf nodes.

[0216] In one exemplary solution, the following formula can be used:

[0217]

[0218] Where, d H (p i ,p j ) represents the distance between two different prototype feature vectors.

[0219] In this embodiment, it is also proposed to reduce the similarity between prototype feature vectors represented by leaf nodes by widening the distance between any two leaf nodes, thereby avoiding the situation where the prototype feature vectors are too clustered and cannot effectively represent all the features of the training image, so that the training image can be fully classified based on the prototype feature vectors.

[0220] When updating the prototype feature vectors of multiple categories and M spatial feature vectors, it is possible to base it on the aforementioned L II-feat L II-dist L PI L PP-LCA L P-Origin and L PP-leaf At least one loss value is achieved.

[0221] It is understandable that in the specific implementation of the training method for the image recognition model, the aforementioned multiple losses can be directly summed, or different weights can be configured for the aforementioned multiple losses to achieve the training of the image recognition model; no restrictions are imposed here.

[0222] S104. Determine the predicted category of the training image based on the similarity between the feature vector of the training image and the prototype features of multiple categories.

[0223] Step S104 is described above. Figure 3 The classification module shown executes the classification model, which includes a classifier. The classifier analyzes the similarity between the feature vectors of the training image and the prototype features of multiple categories to predict the category of the training image.

[0224] Specifically, by adaptively adjusting weights, a similarity mapping is performed between the feature vectors of the training image and the prototype features of multiple categories to obtain the predicted category of the training image:

[0225] Where W is the weight of the similarity between the feature vector of the training image and each prototype feature vector in the prototype feature vectors of multiple categories, S(z i ) represents the similarity between the feature vector of the training image and the prototype features of multiple categories, and σ(·) is the activation function.

[0226] For example, for a training image's feature vector Q1, there are multiple class-specific prototype feature vectors including prototype feature vectors Y1, Y2, ..., Yz. The classifier calculates the similarity between the training image's feature vector Q1 and each of the prototype feature vectors Y1, Y2, ..., Yz, where W represents the weight of each prototype feature vector Y1, Y2, ..., Yz. The similarity between the training image's feature vector Q1 and the prototype feature vector Y1 is then weighted using the weights corresponding to the prototype feature vector Y1. Since the multiple class-specific prototype feature vectors include z prototype feature vectors, the similarity between the training image's feature vector Q1 and each of the z prototype feature vectors is calculated separately, and after weighting, these z results are summed to obtain the score corresponding to the training image's feature vector Q1. When the score is greater than or equal to the threshold, the training image is considered a live face image; when the score is less than the threshold, the training image is considered a non-live face image, and no restriction is imposed here.

[0227] Furthermore, since W can be an adaptive weight in the analysis of the predicted categories of the training images, W can be corrected using the following loss function:

[0228]

[0229] Among them, y i To train the true class of the image, σ(W·S(z) i )) represents the predicted category of the training image, σ(·) represents the activation function, and (x i ,y i ) is a pair consisting of the feature vector of the training image and the true class of the training image.

[0230] In this embodiment, the predicted category of the training image is determined by analyzing the similarity between multiple training images and prototype feature vectors of multiple categories, and then combining the weight of each prototype feature vector in the prototype feature vectors of multiple categories. During model training, the predicted category of the training image is analyzed by using the similarity between prototype feature vectors of multiple categories and training images. Since the determination of prototype feature vectors requires the analysis of a tree structure composed of prototype feature vectors of multiple types and M spatial feature vectors, and the M spatial feature vectors fully express the hierarchical structure between prototype feature vectors of multiple types, the hierarchical features of multiple training images are fully utilized, thereby effectively improving the analysis accuracy of the scheme.

[0231] For example, in a face liveness detection scheme, the true category of the training image indicates whether the training sample is a live human image, and the predicted category of the training image indicates whether the training sample is a live human image.

[0232] It is understood that the description of how to determine the predicted category of a training image based on the similarity between the feature vector of the training image and the prototype features of multiple categories is only an example. In actual applications, it should be set according to the specific application scenario, and no restrictions are imposed here.

[0233] In this embodiment, after obtaining the feature vector of the training image, prototype feature vectors of multiple categories, and M spatial feature vectors, the prototype feature vectors of multiple categories are updated based on the similarity between the feature vector of the training image and the prototype feature vectors of multiple categories. The prototype feature vectors of multiple categories and the M spatial feature vectors form a tree structure. Each category has one or more prototype feature vectors, represented by leaf nodes in the tree structure, and the spatial feature vectors by non-leaf nodes. M is a positive integer. Based on the relationship between leaf nodes (prototype feature vectors) and non-leaf nodes (spatial feature vectors) in the tree structure, the prototype feature vectors of multiple categories and the M spatial feature vectors are updated. The updated prototype feature vectors of multiple categories are used to determine the category of the target image. By constructing a tree structure between the prototype feature vectors of multiple categories and the M spatial feature vectors, a more effective feature space representation is formed. Utilizing this tree structure to update the prototype feature vectors of multiple categories and the M spatial feature vectors reduces feature loss during the learning process, improves the accuracy of the image recognition model, and thus overcomes the accuracy bottleneck of the image recognition model.

[0234] Based on the foregoing Figure 3 The image recognition model structure shown above and the aforementioned Figure 4The training method and image recognition model described in this application can be used in different scenarios, such as verifying the identity of the subject when remotely operating banking business, verifying the identity of the subject as a payment password, verifying the identity of the subject when clocking in and out, or verifying the identity of the subject in an access control system. No restrictions are imposed here.

[0235] Based on the image recognition model obtained using the above training method, the following description uses a terminal device as the execution subject to illustrate this image recognition method. Please refer to [link / reference needed] for details. Figure 8 As shown, one embodiment of the image recognition method includes:

[0236] S201. Obtain the target image;

[0237] The terminal device receives images sent by a third party or images captured by the terminal device itself as the target image.

[0238] Optionally, the type of the target image should be consistent with the type of the training samples in the image recognition model to achieve better analysis results. For example, if the image recognition model is trained using two-dimensional images, then the target image is a two-dimensional image or a two-dimensional image obtained after processing; if the image recognition model is trained using three-dimensional images, then the target image is a three-dimensional image or a three-dimensional image obtained after processing.

[0239] S202. Determine the category of the target image based on the similarity between the target image and the prototype feature vectors of multiple categories.

[0240] The category of the target image is used to describe whether the target image is a live human image or a non-live human image.

[0241] The target image is transmitted to the input network of the image recognition model through the corresponding interface. Then, the category of the target image is determined by using the similarity between the prototype feature vector in the image recognition model and the target image.

[0242] It is understood that the image recognition model and prototype feature vector in this embodiment are adopted using the above-mentioned... Figure 4 The technical solution shown was trained, and the specifics will not be elaborated here.

[0243] The training apparatus for the image recognition model in this application is described in detail below. Please refer to [link / reference]. Figure 9 , Figure 9 This is a schematic diagram of one embodiment of the image processing apparatus in this application. The image processing apparatus 20 includes:

[0244] The acquisition unit 210 is used to acquire the feature vector of the training image, the prototype feature vectors of multiple categories, and M spatial feature vectors. The prototype feature vectors of multiple categories and the M spatial feature vectors form a tree structure. Each category has one or more prototype feature vectors. The prototype feature vectors are represented by the leaf nodes of the tree structure, and the spatial feature vectors are represented by the non-leaf nodes of the tree structure. M is a positive integer.

[0245] The update unit 220 is used to update the prototype feature vectors of multiple categories based on the similarity between the feature vectors of the training image and the prototype feature vectors of multiple categories.

[0246] The update unit 220 is also used to update the prototype feature vectors of multiple categories and M spatial feature vectors based on the relationship between leaf nodes and non-leaf nodes.

[0247] The updated prototype feature vectors of multiple categories are used to determine the category of the target image.

[0248] In this embodiment, after acquiring the feature vector of the training image, the prototype feature vectors of multiple categories, and M spatial feature vectors, the image processing device updates the prototype feature vectors of multiple categories based on the similarity between the feature vector of the training image and the prototype feature vectors of multiple categories. The prototype feature vectors of multiple categories and the M spatial feature vectors form a tree structure. Each category has one or more prototype feature vectors. The prototype feature vectors are represented by leaf nodes in the tree structure, and the spatial feature vectors are represented by non-leaf nodes in the tree structure. M is a positive integer. Based on the relationship between the leaf nodes (prototype feature vectors) and non-leaf nodes (spatial feature vectors) in the tree structure, the prototype feature vectors of multiple categories and the M spatial feature vectors are updated. The updated prototype feature vectors of multiple categories are used to determine the category of the target image. By constructing a tree structure between prototype feature vectors of multiple categories and M spatial feature vectors, a more effective feature space representation is formed. By utilizing the tree structure between prototype feature vectors of multiple categories and M spatial feature vectors, the prototype feature vectors of multiple categories and M spatial feature vectors are updated, reducing feature loss during the learning process, improving the accuracy of the image recognition model, and thus breaking through the accuracy bottleneck of the image recognition model.

[0249] Optionally, update unit 220 is specifically used for:

[0250] Based on the similarity between the feature vector of the training image and the prototype feature vectors of multiple categories, the target prototype feature vector is determined. The target prototype feature vector is the prototype feature vector with the highest similarity to the feature vector of the training image among the prototype feature vectors of multiple categories.

[0251] The target prototype feature vector is updated by increasing the similarity between the feature vector of the training image and the feature vector of the target prototype.

[0252] In this embodiment, the similarity between the feature vector of the training image and the prototype feature vector of multiple categories is used to determine the target prototype feature vector that is most similar to the feature vector of the training image. Then, the target prototype feature vector is updated by increasing the similarity between the feature vector of the training image and the target prototype feature vector. This can effectively update the target prototype feature vector based on the current training feature vector, so that the updated target prototype feature vector can more accurately classify the feature vector of the training image.

[0253] Optionally, the update unit 220 is also used to update the prototype feature vectors of multiple categories by reducing the difference between the first similarity and the second similarity. The first similarity is the similarity between the feature vector of the training image and any one of the prototype feature vectors of the multiple categories. The second similarity is the similarity between the feature vector of the enhanced image and any one of the prototype feature vectors. The enhanced image is obtained by performing data augmentation processing on the training image.

[0254] In this embodiment of the application, in order to ensure that the enhanced image obtained by feature enhancement of the training image is similar to the training image, the prototype feature vectors of multiple categories are updated by reducing the difference between the first similarity and the second similarity. This makes the similarity between the feature vectors corresponding to the enhanced image and the training image and any prototype feature vector of the multiple categories consistent, thus avoiding the situation where the distribution of the obtained enhanced image and the training image in the vector space is inconsistent.

[0255] Optionally, update unit 220 is specifically used for:

[0256] The first non-leaf node is determined from the non-leaf nodes. The first non-leaf node represents the nearest common parent node of the leaf node of the first prototype feature vector and the leaf node of the second prototype feature vector. The first prototype feature vector is the K nearest neighbor feature vector of the second prototype feature vector, and the second prototype feature vector is the K nearest neighbor feature vector of the first prototype feature vector. Both the first prototype feature vector and the second prototype feature vector belong to the prototype feature vector, and K is a positive integer.

[0257] By increasing the similarity between the first prototype feature vector and the spatial feature vector represented by the first non-leaf node, the prototype feature vectors of multiple categories and M spatial feature vectors are updated.

[0258] In this embodiment, when the first non-leaf node is the nearest common parent node of the leaf nodes corresponding to the first prototype feature vector and the second prototype feature vector, and the first prototype feature vector and the second prototype feature vector are K nearest neighbor feature vectors, the similarity between the first prototype feature vector and the first non-leaf node (first spatial feature vector) is improved, providing a direction for reasonably and effectively updating the M spatial feature vectors.

[0259] Optionally, update unit 220 is also used for:

[0260] The second non-leaf node is determined from the non-leaf nodes. The second non-leaf node is the nearest common parent node of the leaf node representing the first prototype feature vector, the leaf node representing the second prototype feature vector, and the leaf node representing the third prototype feature vector. The third prototype feature vector does not belong to the K nearest neighbor feature vector of the first prototype feature vector, but belongs to the K nearest neighbor feature vector of the second prototype feature vector.

[0261] By reducing at least one of the following similarities: the similarity between the first prototype feature vector and the spatial feature vector represented by the second non-leaf node, the similarity between the second prototype feature vector and the spatial feature vector represented by the second non-leaf node, and the similarity between the third prototype feature vector and the spatial feature vector represented by the first non-leaf node, the prototype feature vector of multiple categories and M spatial feature vectors are updated.

[0262] In this embodiment, when the second non-leaf node is the nearest common parent node of the leaf nodes corresponding to the first, second, and third prototype feature vectors, the first and second prototype feature vectors are each other's K nearest neighbor feature vectors. When the third prototype feature vector and the second prototype feature vector are not each other's K nearest neighbor feature vectors, the similarity between the first and second prototype feature vectors and the second leaf node is reduced, or the similarity between the third prototype feature vector and the first leaf node is reduced. This is to make the prototype feature vectors represented by the leaf nodes far away from the spatial feature vectors represented by the non-leaf nodes, so that among the representations of multiple categories of prototype feature vectors and M spatial feature vectors, the prototype feature vectors corresponding to the parent nodes of the leaf nodes can have richer expressions.

[0263] Optionally, the update unit 220 is also used to update the prototype feature vectors of multiple categories and M spatial feature vectors by reducing the similarity between the prototype feature vectors represented by the leaf nodes and the spatial feature vectors represented by the root node, where the root node is one of the non-leaf nodes.

[0264] In this embodiment of the application, it is also proposed that by widening the distance between the prototype feature vector represented by the leaf node and the spatial feature vector represented by the root node, the similarity between the prototype feature vector represented by the leaf node and the spatial feature vector represented by the root node can be reduced, thereby enabling M spatial feature vectors to have a richer expression when they have the same number of levels.

[0265] Optionally, the update unit 220 is also used to update the prototype feature vectors of multiple categories and M spatial feature vectors by reducing the similarity between the prototype feature vectors represented by any two leaf nodes.

[0266] In this embodiment, it is also proposed to reduce the similarity between prototype feature vectors represented by leaf nodes by widening the distance between any two leaf nodes, thereby avoiding the situation where the prototype feature vectors are too clustered and cannot effectively represent all the features of the training image, so that the training image can be fully classified based on the prototype feature vectors.

[0267] Optionally, update unit 220 is specifically used for:

[0268] Determine the predicted probability of each non-leaf node, where the predicted probability is the probability that the non-leaf node is the nearest common parent node of the leaf node representing the first prototype feature vector and the leaf node representing the second prototype feature vector.

[0269] Determine the sampling result for each non-leaf node, where the sampling result is the sum of the predicted probability and the random probability;

[0270] The first non-leaf node is determined from the non-leaf nodes based on the sampling results of the non-leaf nodes.

[0271] In this embodiment, a predicted probability for each non-leaf node is introduced. Sampling is performed from the prototype feature vectors corresponding to multiple non-leaf nodes. The probability that the sampled leaf node is the first leaf node is the predicted probability. To avoid the risk of overfitting from multiple samplings, which could result in only a fixed number of non-leaf nodes being sampled, a random probability is introduced as a perturbation. The sum of the predicted probability and the random probability of the non-leaf node is used to represent whether the sampled non-leaf node is the first leaf node, thus reducing the probability of overfitting during sampling. This avoids the situation where only a few fixed non-leaf nodes are sampled during the sampling of the first non-leaf node, preventing a comprehensive update of the M spatial feature vectors and multiple prototype feature vectors.

[0272] Optionally, the update unit 220 is also used to update the feature vectors of the training image and the feature vectors of the enhanced image by increasing the similarity between the feature vectors of the enhanced image and the feature vectors of the training image.

[0273] In this embodiment, by increasing the similarity between the feature vector of the training image and the feature vector of the enhanced image, it provides strong support for obtaining the enhanced image by feature enhancement of the training image while retaining the features of the training image, and avoids the situation of losing the key features of the training image during the feature enhancement process.

[0274] Optionally, the acquisition unit 210 is also used to acquire the true category of the training image;

[0275] The image analysis device also includes an analysis unit 230, which is further configured to use a classifier to analyze the similarity between the feature vector of the training image and the prototype feature vector of the plurality of categories, and determine the predicted category of the training image.

[0276] The update unit 220 is also used to update the classifier based on the true category of the training image and the predicted category of the training image.

[0277] In this embodiment, the predicted category corresponding to the training image is determined by analyzing the similarity between multiple training images and prototype feature vectors of multiple categories. During model training, the predicted category of the training image is analyzed by utilizing the similarity between prototype feature vectors of multiple categories and training images. Since the determination of prototype feature vectors requires analysis using a tree structure composed of prototype feature vectors of multiple types and M spatial feature vectors, and the M spatial feature vectors fully express the hierarchical structure between prototype feature vectors of multiple types, the hierarchical features of multiple training images are fully utilized, thereby effectively improving the analysis accuracy of the scheme.

[0278] The image recognition device in this application is described in detail below. Please refer to [link / reference]. Figure 10 , Figure 10 This is a schematic diagram of one embodiment of the image recognition device in this application. The image recognition device 30 includes:

[0279] Acquisition unit 310 is used to acquire the target image;

[0280] Analysis unit 320 is used to determine the category of the target image based on the similarity between the target image and the prototype feature vectors of multiple categories, wherein the prototype feature vectors of the multiple categories are determined based on the aforementioned... Figure 4 The image processing method described in the article is used to obtain the image.

[0281] The image recognition device provided in this application can be used on a server; please refer to [link / reference]. Figure 11 , Figure 11This is a schematic diagram of a server structure provided in an embodiment of this application. The server 400 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 422 (e.g., one or more processors) and memory 432, and one or more storage media 430 (e.g., one or more mass storage devices) for storing application programs 442 or data 444. The memory 432 and storage media 430 can be temporary or persistent storage. The program stored in the storage media 430 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the CPU 422 may be configured to communicate with the storage media 430 and execute a series of instruction operations in the storage media 440 on the server 400.

[0282] Server 400 may also include one or more power supplies 426, one or more wired or wireless network interfaces 450, one or more input / output interfaces 458, and / or one or more operating systems 441, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0283] The steps performed by the server in the above embodiments can be based on this Figure 11 The server structure shown.

[0284] The image recognition device provided in this application can be used in terminal devices. Please refer to [link / reference]. Figure 12 For ease of explanation, only the parts relevant to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. In the embodiments of this application, a smartphone is used as an example for illustration:

[0285] Figure 12 This is a block diagram illustrating a portion of the structure of a smartphone related to the terminal device provided in the embodiments of this application. (Reference) Figure 12 The smartphone includes components such as a radio frequency (RF) circuit 510, a memory 520, an input unit 530, a display unit 540, a sensor 550, an audio circuit 560, a wireless fidelity (WiFi) module 570, a processor 580, and a power supply 590. Those skilled in the art will understand that... Figure 12 The smartphone structure shown does not constitute a limitation on smartphones and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0286] The following is combined with Figure 12 A detailed introduction to the various components of a smartphone:

[0287] RF circuit 510 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with processor 580; additionally, it transmits uplink data to the base station. Typically, RF circuit 510 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, RF circuit 510 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Message Service (SMS), etc.

[0288] The memory 520 can be used to store software programs and modules. The processor 580 executes various functions and data processing of the smartphone by running the software programs and modules stored in the memory 520. The memory 520 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the smartphone (such as audio data, phonebook, etc.). In addition, the memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0289] The input unit 530 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the smartphone. Specifically, the input unit 530 may include a touch panel 531 and other input devices 532. The touch panel 531, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 531), and drive corresponding connected devices according to a pre-set program. Optionally, the touch panel 531 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 580, and can receive and execute commands sent by the processor 580. In addition, the touch panel 531 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 531, the input unit 530 may also include other input devices 532. Specifically, other input devices 532 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0290] Display unit 540 can be used to display information input by the user or information provided to the user, as well as various menus of the smartphone. Display unit 540 may include display panel 541, optionally configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar form. Further, touch panel 531 may cover display panel 541. When touch panel 531 detects a touch operation on or near it, it transmits the information to processor 580 to determine the type of touch event. Subsequently, processor 580 provides corresponding visual output on display panel 541 based on the type of touch event. Although in Figure 12 In this embodiment, the touch panel 531 and the display panel 541 are two separate components to realize the input and output functions of the smartphone. However, in some embodiments, the touch panel 531 and the display panel 541 can be integrated to realize the input and output functions of the smartphone.

[0291] The smartphone may also include at least one sensor 550, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 541 according to the ambient light level, and the proximity sensor can turn off the display panel 541 and / or backlight when the smartphone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the smartphone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, tapping), etc. Other sensors that may be configured in the smartphone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0292] Audio circuit 560, speaker 561, and microphone 562 provide an audio interface between the user and the smartphone. Audio circuit 560 converts received audio data into electrical signals and transmits them to speaker 561, where speaker 561 converts them into sound signals for output. On the other hand, microphone 562 converts collected sound signals into electrical signals, which are received by audio circuit 560, converted into audio data, and then processed by processor 580 before being transmitted via RF circuit 510 to, for example, another smartphone, or the audio data can be output to memory 520 for further processing.

[0293] WiFi is a short-range wireless transmission technology. Smartphones, through their WiFi modules (570), can help users send and receive emails, browse web pages, and access streaming media, providing wireless broadband internet access. Although Figure 12 WiFi module 570 is shown, but it is understood that it is not an essential component of a smartphone and can be omitted as needed without changing the nature of the invention.

[0294] The processor 580 is the control center of the smartphone, connecting various parts of the smartphone through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 520, and by calling data stored in the memory 520, thereby providing overall monitoring of the smartphone. Optionally, the processor 580 may include one or more processing units; optionally, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the aforementioned modem processor may also not be integrated into the processor 580.

[0295] The smartphone also includes a power supply 590 (such as a battery) that powers various components. Optionally, the power supply can be logically connected to the processor 580 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0296] Although not shown, smartphones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0297] The steps performed by the terminal device in the above embodiments can be based on this Figure 12 The terminal device structure is shown.

[0298] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the foregoing embodiments.

[0299] This application also provides a computer program product including a program, which, when run on a computer, causes the computer to perform the methods described in the foregoing embodiments.

[0300] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0301] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0302] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0303] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0304] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0305] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized in that, include: The training image's feature vector, multiple category prototype feature vectors, and M spatial feature vectors are obtained. The multiple category prototype feature vectors and the M spatial feature vectors form a tree structure. Each category has one or more prototype feature vectors. The prototype feature vectors are represented by the leaf nodes of the tree structure, and the spatial feature vectors are represented by the non-leaf nodes of the tree structure. M is a positive integer. The prototype feature vectors of the multiple categories are updated based on the similarity between the feature vectors of the training images and the prototype feature vectors of the multiple categories. A first non-leaf node is determined from the non-leaf nodes. The first non-leaf node represents the nearest common parent node of the leaf node of the first prototype feature vector and the leaf node of the second prototype feature vector. The first prototype feature vector is the K nearest neighbor feature vector of the second prototype feature vector, and the second prototype feature vector is the K nearest neighbor feature vector of the first prototype feature vector. Both the first prototype feature vector and the second prototype feature vector belong to the prototype feature vector, and K is a positive integer. By increasing the similarity between the first prototype feature vector and the spatial feature vector represented by the first non-leaf node, the prototype feature vectors of the multiple categories and the M spatial feature vectors are updated. The updated prototype feature vectors of the multiple categories are used to determine the category of the target image.

2. The method according to claim 1, characterized in that, The step of updating the prototype feature vectors of the multiple categories based on the similarity between the feature vectors of the training image and the prototype feature vectors of the multiple categories includes: Based on the similarity between the feature vector of the training image and the prototype feature vectors of the multiple categories, a target prototype feature vector is determined. The target prototype feature vector is the prototype feature vector among the prototype feature vectors of the multiple categories that has the highest similarity to the feature vector of the training image. The target prototype feature vector is updated by increasing the similarity between the feature vector of the training image and the feature vector of the target prototype.

3. The method according to claim 2, characterized in that, The method further includes: By narrowing the difference between the first similarity and the second similarity, the prototype feature vectors of the multiple categories are updated. The first similarity is the similarity between the feature vector of the training image and any one of the prototype feature vectors of the multiple categories. The second similarity is the similarity between the feature vector of the enhanced image and any one of the prototype feature vectors. The enhanced image is obtained by performing data augmentation processing on the training image.

4. The method according to claim 1, characterized in that, The method further includes: A second non-leaf node is determined from the non-leaf nodes. The second non-leaf node is the nearest common parent node of the leaf node representing the first prototype feature vector, the leaf node representing the second prototype feature vector, and the leaf node representing the third prototype feature vector. The third prototype feature vector does not belong to the K nearest neighbor feature vector of the first prototype feature vector, but belongs to the K nearest neighbor feature vector of the second prototype feature vector. The prototype feature vectors of the plurality of categories and the M spatial feature vectors are updated by reducing at least one of the following similarities: the similarity between the first prototype feature vector and the spatial feature vector represented by the second non-leaf node, the similarity between the second prototype feature vector and the spatial feature vector represented by the second non-leaf node, and the similarity between the third prototype feature vector and the spatial feature vector represented by the first non-leaf node.

5. The method according to claim 1, characterized in that, The method further includes: By reducing the similarity between the prototype feature vectors represented by the leaf nodes and the spatial feature vectors represented by the root node, the prototype feature vectors of the multiple categories and the M spatial feature vectors are updated, where the root node is one of the non-leaf nodes.

6. The method according to claim 1, characterized in that, The method further includes: The prototype feature vectors of the multiple categories and the M spatial feature vectors are updated by reducing the similarity between the prototype feature vectors represented by any two leaf nodes.

7. The method according to claim 1, characterized in that, Determining the first non-leaf node from the non-leaf nodes includes: Determine the predicted probability of each non-leaf node, where the predicted probability is the probability that the non-leaf node is the nearest common parent node of the leaf node representing the first prototype feature vector and the leaf node representing the second prototype feature vector. Determine the sampling result for each non-leaf node, where the sampling result is the sum of the predicted probability and the random probability; The first non-leaf node is determined from the non-leaf nodes based on the sampling results of the non-leaf nodes.

8. The method according to claim 3, characterized in that, The method further includes: The feature vectors of the training image and the enhanced image are updated by increasing the similarity between the feature vectors of the enhanced image and the feature vectors of the training image.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: Obtain the true category of the training image; The similarity between the feature vector of the training image and the prototype feature vector of the multiple categories is analyzed using a classifier to determine the predicted category of the training image; The classifier is updated based on the true category and the predicted category of the training images.

10. An image recognition method, characterized in that, include: Acquire the target image; The category of the target image is determined based on the similarity between the target image and the prototype feature vectors of multiple categories, wherein the prototype feature vectors of the multiple categories are obtained by the method according to any one of claims 1 to 9.

11. An image processing apparatus, characterized in that, include: The acquisition unit is used to acquire feature vectors of training images, prototype feature vectors of multiple categories, and M spatial feature vectors. The prototype feature vectors of multiple categories and the M spatial feature vectors form a tree structure. Each category has one or more prototype feature vectors. The prototype feature vectors are represented by the leaf nodes of the tree structure, and the spatial feature vectors are represented by the non-leaf nodes of the tree structure. M is a positive integer. An update unit is used to update the prototype feature vectors of the multiple categories based on the similarity between the feature vectors of the training image and the prototype feature vectors of the multiple categories. The updating unit is further configured to determine a first non-leaf node from the non-leaf nodes, wherein the first non-leaf node represents the nearest common parent node of the leaf node of the first prototype feature vector and the leaf node of the second prototype feature vector, the first prototype feature vector is the K nearest neighbor feature vector of the second prototype feature vector, the second prototype feature vector is the K nearest neighbor feature vector of the first prototype feature vector, both the first prototype feature vector and the second prototype feature vector belong to the prototype feature vector, and K is a positive integer; by increasing the similarity between the first prototype feature vector and the spatial feature vector represented by the first non-leaf node, the prototype feature vectors of the multiple categories and the M spatial feature vectors are updated; The updated prototype feature vectors of the multiple categories are used to determine the category of the target image.

12. The apparatus according to claim 11, characterized in that, The update unit is further configured to: A second non-leaf node is determined from the non-leaf nodes. The second non-leaf node is the nearest common parent node of the leaf node representing the first prototype feature vector, the leaf node representing the second prototype feature vector, and the leaf node representing the third prototype feature vector. The third prototype feature vector does not belong to the K nearest neighbor feature vector of the first prototype feature vector, but belongs to the K nearest neighbor feature vector of the second prototype feature vector. The prototype feature vectors of the plurality of categories and the M spatial feature vectors are updated by reducing at least one of the following similarities: the similarity between the first prototype feature vector and the spatial feature vector represented by the second non-leaf node, the similarity between the second prototype feature vector and the spatial feature vector represented by the second non-leaf node, and the similarity between the third prototype feature vector and the spatial feature vector represented by the first non-leaf node.

13. An image recognition device, characterized in that, include: The acquisition unit is used to acquire the target image; An analysis unit is configured to determine the category of the target image based on the similarity between the target image and the prototype feature vectors of multiple categories, wherein the prototype feature vectors of the multiple categories are obtained by the method according to any one of claims 1 to 9.

14. A computer device, characterized in that, include: Memory, processor, and bus system; The memory is used to store programs; The processor is configured to execute a program in the memory, and the processor is configured to execute the method of any one of claims 1 to 9 or claim 10 according to the instructions in the program code; The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

15. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method as claimed in any one of claims 1 to 9 or 10.

16. A computer program product, comprising a computer program, characterized in that, The computer program is executed by a processor using the method as described in any one of claims 1 to 9 or 10.

Images