Portrait clustering method and device based on sparse regularization decoupling non-local attention

Abstract

The application discloses a portrait clustering method and device based on sparse regularization decoupling non-local attention, and relates to the field of image processing, and comprises the following steps: constructing a portrait clustering model and training to obtain a trained portrait clustering model; a sparse regularization decoupling non-local attention module in the model receives output features from a previous structure as input features, and generates query features, key features, value features and unary features respectively; a pair-wise attention weight matrix for representing the similarity between the query features and the key features is generated through a query-related pair-wise context stream; a unary attention weight matrix is generated by processing the unary features through a query-irrelevant unary context stream; and sparse regularization decoupled features are generated based on the pair-wise attention weight matrix, the unary attention weight matrix, the value features and the input features. The application solves the problem that non-local attention is difficult to effectively and independently learn pair-wise relationship item context information and unary item context information.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a portrait clustering method and device based on sparse regularization decoupling non-local attention. BACKGROUND

[0002] Traditional convolutional neural networks have inherent limitations in capturing long-range dependencies in data and modeling global context information due to their core operation based on local receptive fields. To effectively overcome this limitation, the non-local attention mechanism is introduced into the visual field, aiming to establish global associations between elements in the data, thereby providing a new way for more comprehensive feature learning and fusion.

[0003] The calculation process of the existing standard non-local attention mechanism, although excellent in modeling global context, actually couples two different types of context information internally. One is the pair-wise relationship context information, which is usually represented as the pair-wise relationship between different pixel positions, and it captures the mutual connection between specific elements. The other is the unary context information, which is usually represented as the inherent saliency or universal influence of a single pixel or region itself, but this information is irrelevant to the query position itself in many cases. However, the close coupling of these two types of context information in the non-local attention mechanism makes it difficult for the network to effectively and independently learn different types of visual cues, limiting the potential of the non-local attention mechanism in extracting more discriminative features and thus restricting the overall performance of the model in a wide range of machine vision tasks. SUMMARY

[0004] The present application aims to propose a portrait clustering method and device based on sparse regularization decoupling non-local attention to solve the above-mentioned technical problems.

[0005] In a first aspect, the present application provides a portrait clustering method based on sparse regularization decoupling non-local attention, comprising the following steps:

[0006] The portrait clustering model is constructed and trained to obtain a trained portrait clustering model, and a portrait clustering database is constructed. The portrait clustering model includes a residual convolution backbone network and a generalized average pooling layer connected in sequence, and a sparse regularization decoupled non-local attention module. The sparse regularization decoupled non-local attention module receives the output feature from the previous structure as the input feature, and generates query features, key features, value features and unary features according to the input features. The sparse regularization decoupled non-local attention module decouples the traditional non-local attention mechanism into two parallel query-related pair-wise context streams and query-independent unary context streams. The pair-wise attention weight matrix representing the similarity between the query features and the key features is generated through the query-related pair-wise context stream. The unary attention weight matrix is generated by processing the unary features through the query-independent unary context stream. The sparse regularization decoupled features are generated based on the pair-wise attention weight matrix, the unary attention weight matrix, the value features and the input features, and are used as the output features. The group sparse regularization term is constructed based on the pair-wise attention weight matrix and is used as an optimization constraint in the portrait clustering model training process.

[0007] The portrait image of the person to be queried is obtained and input into the trained portrait clustering model, and the portrait features of the person to be queried are extracted. The similarity between the portrait features of the person to be queried and each portrait feature in the portrait clustering database is calculated to obtain a similarity result, and the portrait image of the person to be queried is determined based on the similarity result.

[0008] As preferred, the query-related pair-wise context stream includes a query whitening module and a key whitening module. In the query-related pair-wise context stream, the query features and the key features are input into the query whitening module and the key whitening module for whitening processing to generate whitened query features and whitened key features, as shown in the following formula:

[0009] ;

[0010] ;

[0011] Wherein, represents the query features, C represents the number of channels, H and W represent the height and width, respectively, represents a set of real numbers; represents the whitened query features; represents a function corresponding to the query whitening module; represents the proportion of batch normalization and instance normalization in the whitening process of the query features; and represent the mean and standard deviation of the query features calculated according to each instance and each channel, respectively; and respectively represent the learnable scale and shift parameters for instance normalization of the query feature; and respectively represent the mean and standard deviation of the query feature computed per channel over the whole batch; and respectively represent the learnable scale and shift parameters for batch normalization of the query feature; represents a matrix multiplication operation; represents an element-wise addition operation; represents a key feature; represents a whitened key feature; represents a function corresponding to the key whitening module; represents a control of the proportion of instance normalization and batch normalization in the whitening process of the key feature; and respectively represent the mean and standard deviation of the key feature computed per instance, per channel; and respectively represent the learnable scale and shift parameters for instance normalization of the key feature; and respectively represent the mean and standard deviation of the key feature computed per channel over the whole batch; and respectively represent the learnable scale and shift parameters for batch normalization of the key feature;

[0012] The whitened query feature and the whitened key feature are reshaped respectively to obtain a reshaped whitened query feature and a reshaped whitened key feature. After performing matrix multiplication on the reshaped whitened query feature and the reshaped whitened key feature and applying a Softmax function, a pair-wise attention weight matrix is calculated, which specifically includes:

[0013] ;

[0014] wherein, represents a pair-wise attention weight matrix; Reshape represents a reshaping operation; represents performing normalization on each row by executing function.

[0015] As a preferred, in the query-independent unary context flow, the unary feature is first reshaped, and then the Softmax function and the broadcast operation are applied, as shown in the following formula:

[0016] ;

[0017] wherein, represents a unary attention weight matrix; Reshape denotes reshaping operation; Reshape denotes reshaping operation Reshape denotes reshaping operation C denotes the number of channels, and H and W denote height and width, respectively, Reshape denotes reshaping operation

[0018] As a preferred, the sparse regularization decoupling feature is generated based on the pair-wise attention weight matrix, the unary attention weight matrix, the value feature and the input feature, and is taken as the output feature, specifically comprising:

[0019] Reshape denotes reshaping operation

[0020] Reshape denotes reshaping operation

[0021] Reshape denotes reshaping operation

[0022] Reshape denotes reshaping operation

[0023] ;

[0024] wherein, Reshape denotes reshaping operation C denotes the number of channels, and H and W denote height and width, respectively, Reshape denotes reshaping operation

[0025] As a preferred, the group sparse regularization term is represented as:

[0026]

[0027] wherein, Reshape denotes reshaping operation Reshape denotes reshaping operation Reshape denotes reshaping operation denotes the L2,1 norm of the pair-wise attention weight matrix, i.e., summing the L2 norm of each row of the pair-wise attention weight matrix; denotes the element in the mth row and the nth column of the pair-wise attention weight matrix; denotes the L2 norm of the mth row in the pair-wise attention weight matrix.

[0028] As preferred, the residual convolution backbone network comprises a stem and a residual group connected in sequence in the ResNet50; the residual group comprises a first residual group, a second residual group, a third residual group and a fourth residual group connected in sequence; the sparse regularization decoupled non-local attention module is embedded between the second residual group and the third residual group and between the third residual group and the fourth residual group, respectively.

[0029] In the training phase of the portrait profiling model, the group sparse regularization term is combined with the main loss function of the portrait profiling model in a weighted manner to form a total loss function, and the portrait profiling model is trained based on the total loss function.

[0030] In a second aspect, the present application provides a portrait profiling device based on sparse regularization decoupled non-local attention, comprising:

[0031] The model construction module is configured to construct and train the portrait profiling model to obtain the trained portrait profiling model, and to construct the portrait profiling database; the portrait profiling model comprises a residual convolution backbone network and a generalized average pooling layer, which are connected in sequence and embedded with the sparse regularization decoupled non-local attention module; the sparse regularization decoupled non-local attention module receives the output feature from the previous structure as the input feature, and generates query feature, key feature, value feature and unary feature according to the input feature; the sparse regularization decoupled non-local attention module decouples the traditional non-local attention mechanism into two parallelly arranged query-related pair-wise context stream and query-independent unary context stream; the pair-wise attention weight matrix representing the similarity between the query feature and the key feature is generated through the query-related pair-wise context stream; the unary attention weight matrix is generated by processing the unary feature through the query-independent unary context stream; the sparse regularization decoupled feature is generated based on the pair-wise attention weight matrix, the unary attention weight matrix, the value feature and the input feature and is taken as the output feature; the group sparse regularization term is constructed based on the pair-wise attention weight matrix and is taken as an optimization constraint in the training process of the portrait profiling model.

[0032] The profiling module is configured to obtain the portrait image of the person to be queried and input it into the trained portrait profiling model, extract the portrait feature of the person to be queried, calculate the similarity between the portrait feature of the person to be queried and each portrait feature in the portrait profiling database to obtain the similarity result, and determine the profiling result of the portrait image of the person to be queried based on the similarity result.

[0033] In a third aspect, the present application provides an electronic device, comprising one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation manner of the first aspect.

[0034] In a fourth aspect, the present application provides a computer readable storage medium, having stored thereon a computer program, when the computer program is executed by a processor, the method as described in any implementation manner of the first aspect is implemented.

[0035] In a fifth aspect, the present application provides a computer program product, comprising a computer program, when the computer program is executed by a processor, the method as described in any implementation manner of the first aspect is implemented.

[0036] Compared with the prior art, the present application has the following beneficial effects:

[0037] (1) The portrait clustering method based on sparse regularization decoupling non-local attention proposed in the present application decouples the traditional non-local attention mechanism, so that the pair relationship item and the unary item can independently and more effectively learn the context information they focus on; at the same time, a group sparsification regularization term is introduced for constraint to guide the attention mechanism to focus on discriminative local detail information.

[0038] (2) The portrait clustering method based on sparse regularization decoupling non-local attention proposed in the present application can effectively improve the learning quality of different types of context information, and can improve the visual attention effect and performance in the portrait clustering task. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0040] Figure 1 The flowchart of the portrait clustering method based on sparse regularization decoupling non-local attention of the embodiments of the present application;

[0041] Figure 2 The schematic diagram of the sparse regularization decoupling non-local attention module of the portrait clustering method based on sparse regularization decoupling non-local attention of the embodiments of the present application;

[0042] Figure 3 The schematic diagram of the training process of the portrait clustering method based on sparse regularization decoupling non-local attention of the embodiments of the present application;

[0043] Figure 4 A schematic diagram of a portrait clustering device based on sparse regularization decoupling non-local attention according to an embodiment of the present application;

[0044] Figure 5 A schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0046] Figure 1 A portrait clustering method based on sparse regularization decoupling non-local attention according to an embodiment of the present application is shown, comprising the following steps:

[0047] S1, a portrait clustering model is constructed and trained to obtain a trained portrait clustering model, and a portrait clustering database is constructed; the portrait clustering model comprises a residual convolution backbone network and a generalized average pooling layer connected in sequence, and a sparse regularization decoupling non-local attention module; the sparse regularization decoupling non-local attention module receives output features from the previous structure as input features, and generates query features, key features, value features and unary features according to the input features; the sparse regularization decoupling non-local attention module decouples the traditional non-local attention mechanism into two parallelly arranged query-related pair-wise context streams and query-independent unary context streams; a pair-wise attention weight matrix representing the similarity between the query features and the key features is generated through the query-related pair-wise context streams; the unary attention weight matrix is generated by processing the unary features through the query-independent unary context streams; the sparse regularization decoupled features are generated based on the pair-wise attention weight matrix, the unary attention weight matrix, the value features and the input features, and are used as output features; and a group sparse regularization term is constructed based on the pair-wise attention weight matrix and is used as an optimization constraint in the portrait clustering model training process.

[0048] In a specific embodiment, the query-related pair-wise context stream comprises a query whitening module and a key whitening module, and in the query-related pair-wise context stream, the query features and the key features are input into the query whitening module and the key whitening module for whitening processing to generate whitened query features and whitened key features, as shown in the following formula:

[0049] ;

[0050] ;

[0051] wherein, denotes the query feature, C denotes the number of channels, H and W denote the height and width, respectively, denotes a set of real numbers; denotes the whitened query feature; denotes a function corresponding to the query whitening module; denotes a ratio to control the proportion of batch normalization and instance normalization in the whitening process of the query feature; and denote the mean and standard deviation of the query feature calculated according to each instance and each channel, respectively; and denote the learnable scaling parameter and offset parameter of the instance normalization for the query feature, respectively; and denote the mean and standard deviation of the query feature calculated according to the entire batch and each channel, respectively; and denote the learnable scaling parameter and offset parameter of the batch normalization for the query feature, respectively; denotes a matrix multiplication operation; denotes an element-wise addition operation; denotes the key feature; denotes the whitened key feature; denotes a function corresponding to the key whitening module; denotes a ratio to control the proportion of instance normalization and batch normalization in the whitening process of the key feature; and denote the mean and standard deviation of the key feature calculated according to each instance and each channel, respectively; and denote the learnable scaling parameter and offset parameter of the instance normalization for the key feature, respectively; and denote the mean and standard deviation of the key feature calculated according to the entire batch and each channel, respectively; and denote the learnable scaling parameter and offset parameter of the batch normalization for the key feature, respectively;

[0052] The whitened query feature and the whitened key feature are respectively reshaped to obtain a reshaped whitened query feature and a reshaped whitened key feature; after performing matrix multiplication on the reshaped whitened query feature and the reshaped whitened key feature and applying a Softmax function, a pair-wise attention weight matrix is calculated, which specifically includes:

[0053] ;

[0054] wherein, denotes a reshape operation; denotes performing normalization by row. denotes performing normalization by row.

[0055] In specific embodiments, in the query-independent unary context stream, the unary feature is reshaped first, and then the Softmax function and the broadcast operation are applied, as shown in the following formula:

[0056]

[0057] wherein, denotes a unary attention weight matrix; denotes a unary feature; Reshape denotes a reshape operation; denotes performing normalization by row. denotes performing normalization by row; Expand denotes a broadcast operation for adjusting the shape to the dimension matching the value feature , C denotes the number of channels, H and W respectively denote the height and the width, denotes a set of real numbers.

[0058] Specifically, referring to Figure 2 , the embodiments of the present application propose a sparse regularization decoupled non-local attention module, which first generates query features, key features, value features and unary features from the input features through four independent convolution modules with a convolution kernel size of 1x1, to support parallel processing of different types of feature information, and the convolution kernels of the four convolution modules are respectively denoted as , , and ; then the query whitening module is used to perform mixed normalization processing on the query features; the key whitening module is used to perform mixed normalization processing on the key features. The learnable scaling parameters and offset parameters of the instance normalization of the two modules and the learnable scaling parameters and offset parameters of the batch normalization are independent. Then the whitened query features are reshaped from dimensions to dimensions, and the reshaping process is to perform a feature tensor rearrangement operation; the whitened key features are reshaped from dimensions to dimensions; the reshaped whitened query features and the reshaped whitened key features are multiplied to generate a pair of attention weight matrices representing the similarity between the query features and the key features.

[0059] Further, the unary feature is processed through the query-independent unary context stream to generate a unary attention weight matrix, and the calculation process includes reshaping the unary feature from​ dimension reshaping is dimension, applying a Softmax function and a broadcast operation.

[0060] In specific embodiments, the sparse regularization decoupled feature is generated based on the pair-wise attention weight matrix, the unary attention weight matrix, the value feature and the input feature, and is taken as the output feature, specifically comprising:

[0061] The value feature is reshaped to obtain a reshaped value feature, and the reshaped value feature and the pair-wise attention weight matrix are weighted and summed through matrix multiplication to obtain a first output;

[0062] The reshaped value feature and the unary attention weight matrix are weighted and summed through matrix multiplication to obtain a second output;

[0063] The first output and the second output are respectively reshaped to obtain a reshaped first output and a reshaped second output;

[0064] The reshaped first output and the reshaped second output are fused through element-level addition to obtain a third output; and the third output and the input feature of the sparse regularization decoupled non-local attention module are connected in residual to generate the sparse regularization decoupled feature, as shown in the following formula:

[0065] ;

[0066] wherein, F represents the input feature of the sparse regularization decoupled non-local attention module; F1 represents the reshaped first output; F2 represents the reshaped second output, C represents the number of channels, H and W represent the height and width respectively, represents a set of real numbers.

[0067] Specifically, the first output obtained by weighting the pair-wise attention weight matrix and the value feature and the second output obtained by weighting the unary attention weight matrix and the value feature are element-level added to obtain a third output; and the third output and the input feature of the sparse regularization decoupled non-local attention module are connected in residual to generate the sparse regularization decoupled feature. In the above process, there is also a feature reshaping process, which first needs to reshape the value feature from dimension reshaping is dimension; the first output is reshaped from dimension reshaping is dimension; the second output is reshaped from reshaping is .

[0068] In specific embodiments, the group sparse regularization term is represented as:

[0069]

[0070] wherein, denotes a group sparse regularization term; denotes a pair-wise attention weight matrix, for the pair-wise attention weight matrix, M=N=HW, H and W represent height and width respectively, denotes a set of real numbers; denotes calculating L2,1 norm of the pair-wise attention weight matrix, that is, summing up L2 norm of each row of the pair-wise attention weight matrix; denotes an element in the mth row and the nth column of the pair-wise attention weight matrix; denotes calculating L2 norm of the mth row of the pair-wise attention weight matrix.

[0071] Specifically, the group sparse regularization term is constructed based on the pair-wise attention weight matrix, and the group sparse regularization term is taken as an optimization constraint in the training process of the portrait clustering model, so that the pair-wise attention weight matrix has sparsity.

[0072] An embodiment of the present application designs a sparse regularization decoupled non-local attention mechanism, which can be widely used in the field of computer vision, such as target re-identification, fine-grained identification and other tasks, corresponding to the sparse regularization decoupled non-local attention module. In an embodiment of the present application, it is applied to the portrait clustering task, and therefore a portrait clustering model including a residual convolution backbone network and a generalized average pooling layer connected in sequence with the sparse regularization decoupled non-local attention module is proposed, as shown in Figure 3 .

[0073] Further, the group sparse regularization term is introduced into the main loss function for training the portrait clustering model in a weighted manner to form a total loss function for model training. In one embodiment, the main loss function of the portrait clustering model adopts a cross-entropy loss function, and in other embodiments, the main loss function of the portrait clustering model can select other loss functions. The gradient descent method is used to optimize the total loss function to update the parameters of the model and obtain the trained portrait clustering model.

[0074] In a specific embodiment, the residual convolution backbone network includes a stem and a residual group connected in sequence in ResNet50; the residual group includes a first residual group, a second residual group, a third residual group and a fourth residual group connected in sequence; and the sparse regularization decoupled non-local attention module is embedded between the second residual group and the third residual group and between the third residual group and the fourth residual group.

[0075] In the training phase of the portrait clustering model, the group sparse regularization term is combined with the main loss function of the portrait clustering model in a weighted manner to form a total loss function, and the portrait clustering model is trained based on the total loss function.

[0076] Specifically, the portrait clustering model in the embodiments of the present application is composed of a residual convolutional backbone network ResNet50, a sparsely disentangled non-local attention module (SDNL), and a generalized-mean pooling layer (GeP). In the embodiments, the specific construction manner is a stem of the ResNet50, residual groups: a first residual group (Layer1), a second residual group (Layer2), a third residual group (Layer3), and a fourth residual group (Layer4), the sparsely disentangled non-local attention module (SDNL), and the generalized-mean pooling layer. In one of the embodiments, a sparsely disentangled non-local attention module (SDNL) is embedded after the Layer2 and the Layer3 of the ResNet50 to improve the feature expression capability. Therefore, the output features of the Layer2 and the Layer3 are respectively taken as the input features of the sparsely disentangled non-local attention modules embedded after the Layer2 and the Layer3. In the training process of the portrait clustering model, the group sparsity regularization term is taken as an optimization constraint in the training process of the portrait clustering model, and a cross-entropy loss function is used to construct a total loss function of the portrait clustering model. The portrait clustering model is trained based on the total loss function to obtain a trained portrait clustering model.

[0077] Specifically, the portrait clustering model in the embodiments of the present application is composed of a residual convolutional backbone network ResNet50, a sparsely disentangled non-local attention module (SDNL), and a generalized-mean pooling layer (GeP). In the embodiments, the specific construction manner is a stem of the ResNet50, residual groups: a first residual group (Layer1), a second residual group (Layer2), a third residual group (Layer3), and a fourth residual group (Layer4), the sparsely disentangled non-local attention module (SDNL), and the generalized-mean pooling layer. In one of the embodiments, a sparsely disentangled non-local attention module (SDNL) is embedded after the Layer2 and the Layer3 of the ResNet50 to improve the feature expression capability. Therefore, the output features of the Layer2 and the Layer3 are respectively taken as the input features of the sparsely disentangled non-local attention modules embedded after the Layer2 and the Layer3. In the training process of the portrait clustering model, the group sparsity regularization term is taken as an optimization constraint in the training process of the portrait clustering model, and a cross-entropy loss function is used to construct a total loss function of the portrait clustering model. The portrait clustering model is trained based on the total loss function to obtain a trained portrait clustering model.

[0078] S2, an image of the person to be queried is obtained and input into the trained portrait clustering model, portrait features of the person to be queried are extracted, similarity calculation is performed between the portrait features of the person to be queried and each portrait feature in the portrait clustering database, a similarity result is obtained, and a clustering result of the portrait image of the person to be queried is determined based on the similarity result.

[0079] Specifically, the trained portrait clustering model is deployed, in the inference stage, an image of the person to be queried is obtained and input into the trained portrait clustering model, portrait features of the person to be queried are obtained, similarity between the portrait features of the person to be queried and each portrait feature in the portrait clustering database is calculated, the similarity can be represented by the Euclidean distance, and the pseudo label corresponding to the portrait feature in the portrait clustering database with the nearest Euclidean distance is taken as the clustering result of the portrait image of the person to be queried.

[0080] Further referring to Figure 4 , as an implementation of the method shown in the above figures, the present application provides an embodiment of a portrait clustering device based on sparse regularization decoupling non-local attention, which corresponds to the method embodiment shown in Figure 1 , and the device can be specifically applied to various electronic devices.

[0081] The present application embodiment provides a portrait clustering device based on sparse regularization decoupling non-local attention, which comprises:

[0082] The model construction module 1 is configured to construct a portrait clustering model and train the portrait clustering model to obtain a trained portrait clustering model, and construct a portrait clustering database; the portrait clustering model comprises a residual convolution backbone network and a generalized average pooling layer connected in sequence, and the residual convolution backbone network comprises an embedding sparse regularization decoupling non-local attention module; the sparse regularization decoupling non-local attention module receives output features from the previous structure as input features, generates query features, key features, value features and unary features according to the input features; the sparse regularization decoupling non-local attention module decouples the traditional non-local attention mechanism into two parallelly arranged query-related paired context streams and query-independent unary context streams; the query-related paired context streams are used to generate a paired attention weight matrix for representing the similarity between the query features and the key features; the query-independent unary context streams are used to process the unary features to generate a unary attention weight matrix; the sparse regularization decoupling features are generated based on the paired attention weight matrix, the unary attention weight matrix, the value features and the input features, and are taken as output features; and a group sparse regularization term is constructed based on the paired attention weight matrix and is taken as an optimization constraint in the training process of the portrait clustering model.

[0083] The poly-filing module 2 is configured to acquire a portrait image of a person to be queried and input into a trained portrait poly-filing model, extract portrait features of the person to be queried, perform similarity calculation on the portrait features of the person to be queried and each portrait feature in the portrait poly-filing database, obtain a similarity result, and determine a poly-filing result of the portrait image of the person to be queried based on the similarity result.

[0084] Figure 5 A hardware structure schematic diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 1. Figure 5 As shown in the figure, the electronic device of the embodiment includes a processor 501 and a memory 502; the memory 502 is configured to store computer execution instructions; and the processor 501 is configured to execute the computer execution instructions stored by the memory to implement each step performed by the electronic device in the above-described embodiment. For details, refer to the related description in the foregoing method embodiment.

[0085] Optionally, the memory 502 can be independent or integrated with the processor 501.

[0086] When the memory 502 is independently arranged, the electronic device further includes a bus 503 configured to connect the memory 502 and the processor 501.

[0087] The embodiment of the present application further provides a computer storage medium, and the computer storage medium stores computer execution instructions; when the processor 501 executes the computer execution instructions, the method described above is implemented.

[0088] The embodiment of the present application further provides a computer program product, and the computer program product includes a computer program; when the computer program is executed by the processor 501, the method described above is implemented.

[0089] In the embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the modules is only a logical function division. There can be another division manner in actual implementation, for example, a plurality of modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the modules shown or discussed can be indirect coupling or communication connection through some interfaces, devices or modules, and can be electrical, mechanical or in other forms.

[0090] The modules illustrated as separate components can or can not be physically separate, and the components illustrated as modules can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to implement the embodiment scheme.

[0091] In addition, each functional module in each embodiment of the present application can be integrated in one processing unit, or each module can be physically present alone, or two or more modules can be integrated in one unit. The unit formed by the above modules can be realized in the form of hardware or in the form of hardware plus software functional unit.

[0092] The integrated module realized in the form of software functional module can be stored in a computer readable storage medium. The software functional module is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or the processor 501 to execute part of the steps of the method of each embodiment of the present application.

[0093] It should be understood that the above processor 501 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor, or the processor 501 can also be any conventional processor 501, etc. The steps of the method disclosed in the application can be directly embodied as the execution of the hardware processor 501, or executed by the combination of hardware and software modules in the processor 501.

[0094] The memory 502 can include a high-speed RAM memory, and can also include a non-volatile storage NVM, for example, at least one disk memory, and can also be a U disk, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0095] The bus 503 can be an industry standard architecture (ISA), a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 503 can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus 503 in the drawings of the present application does not limit only one bus 503 or one type of bus 503.

[0096] The aforementioned storage medium can be realized by any type of volatile or nonvolatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0097] An exemplary storage medium is coupled to the processor 501 so that the processor 501 can read information from, and write information to, the storage medium. Of course, the storage medium can be part of the processor 501. The processor 501 and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor 501 and the storage medium can be located in a remote terminal or server.

[0098] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The aforementioned program can be stored in a computer readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes ROM, RAM, magnetic disk or optical disk and various storage media that can store program codes.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for image clustering based on sparse regularization to decouple nonlocal attention, characterized in that, Includes the following steps: A portrait clustering model is constructed and trained to obtain a trained portrait clustering model, and a portrait clustering database is constructed. The portrait clustering model includes a residual convolutional backbone network and a generalized average pooling layer that are sequentially connected and embedded with sparse regularized uncoupled nonlocal attention modules. The sparse regularized uncoupled nonlocal attention module receives the output features from the previous structure as input features and generates query features, key features, value features and unary features according to the input features. The sparse regularized decoupling nonlocal attention module decouples the traditional nonlocal attention mechanism into two parallel query-related pairwise context streams and a query-independent unary context stream; and generates a pairwise attention weight matrix to characterize the similarity between the query features and the key features through the query-related pairwise context streams. The unary features are processed by the query-irrelevant unary context flow to generate a unary attention weight matrix; sparse regularized decoupling features are generated based on the pairwise attention weight matrix, the unary attention weight matrix, the value features, and the input features, and are used as output features; Based on the pairwise attention weight matrix, a group sparse regularization term is constructed and used as an optimization constraint in the training process of the portrait clustering model. The process involves acquiring the portrait image of the person to be queried and inputting it into the trained portrait clustering model. The portrait features of the person to be queried are then extracted. The similarity between the portrait features of the person to be queried and each portrait feature in the portrait clustering database is calculated to obtain the similarity result. Based on the similarity result, the clustering result of the portrait image of the person to be queried is determined.

2. The portrait clustering method based on sparse regularization to decouple nonlocal attention as described in claim 1, characterized in that, The query-related pairwise context stream includes a query whitening module and a key whitening module. In the query-related pairwise context stream, the query features and key features are respectively input into the query whitening module and the key whitening module for whitening processing, generating whitened query features and whitened key features, as shown in the following formula: ； ； in, This represents the query feature, where C represents the number of channels, and H and W represent the height and width, respectively. Represents the set of real numbers; Indicates whitening query characteristics; This indicates the function corresponding to the whitening module; This indicates the ratio of batch normalization to instance normalization during the whitening of query features; and These represent the mean and standard deviation of the query features calculated for each instance and each channel, respectively. and These represent the learnable scaling parameter and offset parameter for instance normalization of the query features, respectively; and These represent the mean and standard deviation of the query features calculated for the entire batch and for each channel, respectively. and These represent the learnable scaling parameter and offset parameter for batch normalization of query features, respectively. Represents matrix multiplication operation; This represents an element-wise addition operation; Indicate key features; Indicates key features of whitening; This represents the function corresponding to the critical whitening module; This represents the ratio of instance normalization to batch normalization during the whitening process of key control features; and These represent the mean and standard deviation of the key features calculated for each instance and each channel, respectively. and These represent the learnable scaling parameter and offset parameter for instance normalization of key features, respectively. and These represent the mean and standard deviation of the key features calculated for the entire batch and for each channel, respectively. and These represent the learnable scaling parameter and offset parameter for batch normalization of key features, respectively. The whitening query feature and whitening key feature are reshaped to obtain the reshaped whitening query feature and the reshaped whitening key feature. After performing matrix multiplication on the reshaped whitening query feature and the reshaped whitening key feature, the Softmax function is applied to calculate the pairwise attention weight matrix, specifically including: ； in, Represents the pairwise attention weight matrix; Reshape represents the reshaping operation; Indicates execution by line. The function is normalized.

3. The portrait clustering method based on sparse regularization to decouple nonlocal attention as described in claim 1, characterized in that, In the query-irrelevant unary context stream, the unary features are first reshaped, and then the Softmax function and broadcast operation are applied, as shown in the following equation: ； in, This represents a univariate attention weight matrix; Represents a univariate feature; Reshape represents a reshaping operation; Indicates execution by line. The function is normalized; Expand represents a broadcast operation used to adjust the shape to match the dimensions of the value features. C represents the number of channels, and H and W represent the height and width, respectively. It represents the set of real numbers.

4. The portrait clustering method based on sparse regularization to decouple nonlocal attention as described in claim 1, characterized in that, Based on the pairwise attention weight matrix, the unary attention weight matrix, the value features, and the input features, sparse regularized decoupling features are generated and used as output features, specifically including: The value features are reshaped to obtain reshaped value features. The reshaped value features are then weighted and summed with the pairwise attention weight matrix by matrix multiplication to obtain the first output. The reshaped value features are weighted and summed with the unary attention weight matrix by matrix multiplication to obtain the second output. The first output and the second output are reshaped respectively to obtain the reshaped first output and the reshaped second output; The reshaped first output and the reshaped second output are fused element-wise to obtain the third output; the third output is then residually concatenated with the input features of the sparse regularized uncoupling nonlocal attention module to generate sparse regularized uncoupling features, as shown in the following equation: ； in, F represents the sparse regularized decoupling feature; F represents the input feature of the sparse regularized decoupling nonlocal attention module; F1 represents the first output after reshaping; F2 represents the second output after reshaping; C represents the number of channels; H and W represent the height and width, respectively. It represents the set of real numbers.

5. The portrait clustering method based on sparse regularization to decouple nonlocal attention as described in claim 1, characterized in that, The group of sparse regularization terms is represented as follows: ； in, This represents the sparse regularization term for the group; This represents the pairwise attention weight matrix. For a pairwise attention weight matrix, M=N=HW, where H and W represent the height and width, respectively. Represents the set of real numbers; This means calculating the L2,1 norm of the pairwise attention weight matrix, which involves calculating the L2 norm of the pairwise attention weight matrix row by row and then summing the results. This represents the element in the m-th row and n-th column of the pairwise attention weight matrix; This indicates that the L2 norm of the m-th row in the pairwise attention weight matrix is ​​calculated.

6. The portrait clustering method based on sparse regularization to decouple nonlocal attention as described in claim 1, characterized in that, The residual convolutional backbone network includes a stem and residual groups connected in sequence in ResNet50; the residual groups include a first residual group, a second residual group, a third residual group, and a fourth residual group connected in sequence; the sparse regularized decoupling nonlocal attention module is embedded between the second residual group and the third residual group and between the third residual group and the fourth residual group, respectively. During the training phase of the portrait clustering model, the group of sparse regularization terms are combined with the main loss function of the portrait clustering model in a weighted manner to form the total loss function, and the portrait clustering model is trained based on the total loss function.

7. A human image clustering device based on sparse regularization to decouple nonlocal attention, characterized in that, include: The model building module is configured to build and train a portrait clustering model to obtain a trained portrait clustering model and build a portrait clustering database. The portrait clustering model includes a residual convolutional backbone network and a generalized average pooling layer that are sequentially connected and embedding a sparse regularized uncoupled nonlocal attention module. The sparse regularized uncoupled nonlocal attention module receives the output features from the previous structure as input features and generates query features, key features, value features and unary features according to the input features. The sparse regularized decoupling nonlocal attention module decouples the traditional nonlocal attention mechanism into two parallel query-related pairwise context streams and a query-independent unary context stream; and generates a pairwise attention weight matrix to characterize the similarity between the query features and the key features through the query-related pairwise context streams. The unary features are processed by the query-irrelevant unary context flow to generate a unary attention weight matrix; sparse regularized decoupling features are generated based on the pairwise attention weight matrix, the unary attention weight matrix, the value features, and the input features, and are used as output features; Based on the pairwise attention weight matrix, a group sparse regularization term is constructed and used as an optimization constraint in the training process of the portrait clustering model. The image aggregation module is configured to acquire the portrait image of the person to be queried and input it into the trained image aggregation model, extract the portrait features of the person to be queried, calculate the similarity between the portrait features of the person to be queried and each portrait feature in the image aggregation database, obtain the similarity result, and determine the image aggregation result of the portrait image of the person to be queried based on the similarity result.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images