A method, apparatus, electronic device and storage medium for pedestrian attribute recognition
By introducing a target attention module and a global max pooling layer into the pedestrian attribute recognition model, the problem of insufficient discriminative power in existing models is solved, and higher attribute recognition accuracy and robustness are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HISENSE GRP HLDG CO LTD
- Filing Date
- 2023-07-31
- Publication Date
- 2026-05-26
AI Technical Summary
Existing pedestrian attribute recognition models, due to their use of global average pooling or average pooling network structures, suffer from poor discriminative power, resulting in unstable attribute recognition accuracy.
An attribute recognition model based on convolutional neural networks and a target attention module is adopted. By adding a global max pooling layer to the bilinear attention neural network structure, second-order statistical information and local nearest neighbor relationships in the image are captured, and more refined features are obtained by utilizing the target attention module.
It improves the discriminative power and accuracy of the attribute recognition model, achieves stable attribute recognition results, and meets the reliability requirements.
Smart Images
Figure CN117237981B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer vision technology, and in particular to a method, apparatus, electronic device and storage medium for pedestrian attribute recognition. Background Technology
[0002] Pedestrian attribute recognition is the process of automatically identifying pedestrian attributes from a given input image containing pedestrians using computer vision algorithms. These attributes include gender, whether the pedestrian wears glasses, age, hair length, whether they carry a backpack, clothing style, and the colors of their upper and lower body. It is of great significance for pedestrian target recognition and pedestrian image retrieval and is widely used in public safety, smart cities, smart transportation, security, and industry.
[0003] With the development of deep learning technology, more and more neural network structures have been proposed and used to handle pedestrian attribute recognition tasks. Currently, most neural network models used for pedestrian attribute recognition use classic convolutional neural network structures as their backbone, constructed using global average pooling (GAP) or AVGPooling. In practical applications, it has been found that when using these models to handle pedestrian attribute recognition tasks, the models exhibit poor discriminative power, leading to unstable attribute recognition accuracy. Summary of the Invention
[0004] This application provides a pedestrian attribute recognition method, device, electronic device, and storage medium to improve the discriminative power of the attribute recognition model and enhance the accuracy and reliability of attribute recognition.
[0005] The specific technical solutions provided in this application are as follows:
[0006] In a first aspect, embodiments of this application provide a pedestrian attribute recognition method, including:
[0007] The input image containing the target object is preprocessed to obtain the image to be identified;
[0008] The image to be identified is input into the attribute recognition model, and based on the attribute recognition model, the predicted probability of the target object having each reference attribute is obtained. The attribute recognition model is constructed based on a convolutional neural network, a target attention module, and a classifier. The target attention module is obtained by adding at least one global max pooling layer to a bilinear attention neural network structure.
[0009] Based on the predicted probabilities of each reference attribute, the attribute recognition result of the target object is obtained.
[0010] Since the attribute recognition model in this application embodiment includes a target attention module, compared with network structures such as GAP or AVGPooling in related technologies, the bilinear attention mechanism can capture second-order statistical information in the image, thereby enabling the attribute recognition model to obtain local nearest neighbor relationships in the image.
[0011] Furthermore, this target attention module is obtained by adding at least one global max pooling layer to the bilinear attention neural network structure. Global max pooling can obtain the maximum value in the feature map of each channel, which allows the attribute recognition model to capture the most relevant elements in the image. This enables the target attention module to obtain more refined features, thereby giving the attribute recognition model better discriminative power. When performing attribute recognition tasks, it can improve the attribute recognition accuracy and robustness while maintaining the attribute recognition accuracy of the stable attribute recognition model, thus meeting the reliability characteristics.
[0012] In one possible implementation, obtaining the predicted probability that the target object possesses each reference attribute based on the attribute recognition model includes:
[0013] Based on the attribute recognition model, the image to be recognized is sequentially processed by multiple mapping modules, processed by a first feature extraction module, and processed by the classifier through global average pooling, fully connected processing, and normalization to obtain the predicted probability of the target object having each reference attribute. Each of the multiple mapping modules is constructed based on the convolutional neural network and the target attention module, and the first feature extraction module is constructed based on the convolutional neural network.
[0014] The above method, because the mapping module includes a target attention module, enables the attribute recognition model to acquire more refined features, thereby improving the discriminative power of the attribute recognition model and increasing the accuracy of attribute recognition.
[0015] In one possible implementation, any mapping module includes a second feature extraction module and the target attention module, wherein the second feature extraction module is constructed based on the convolutional neural network;
[0016] The image to be identified is processed by mapping through any of the mapping modules, including:
[0017] Using the second feature extraction module, the target map is subjected to a first mapping process to obtain a target feature map;
[0018] Using the target attention module, the target feature map is subjected to a second mapping process to obtain an intermediate feature map output by any of the mapping modules;
[0019] Wherein, if any of the mapping modules is the first mapping module in the attribute recognition model, then the target image is the image to be recognized; if any of the mapping modules is not the first mapping module, then the target image is the intermediate feature map output by the preceding mapping module adjacent to any of the mapping modules.
[0020] The purpose of the above method, which performs a second mapping process on the input target feature map through the target attention module, is to adjust the attention of the target feature map to obtain more refined features, thereby making the attribute recognition model more discriminative and improving the accuracy of attribute recognition.
[0021] In one possible implementation, the step of using the target attention module to perform a second mapping process on the target feature map to obtain an intermediate feature map output by any of the mapping modules includes:
[0022] Using the target attention module, perform the following steps:
[0023] The target feature map is subjected to dimensionality reduction and transformation processing to obtain the first feature map;
[0024] The first feature map is subjected to global max pooling and convolution to obtain a first intermediate attention feature map, and the first feature map is subjected to convolution to obtain a second feature map.
[0025] The first intermediate attention feature map and the second feature map are fused to obtain the second intermediate attention feature map, and the target attention feature map is obtained based on the target feature map and the second intermediate attention feature map.
[0026] The target feature map and the target attention feature map are subjected to residual processing to obtain an intermediate feature map output by any of the mapping modules.
[0027] The above method, through global max pooling, fusing the first intermediate attention feature map and the second feature map, and obtaining the target attention feature map based on the target feature map and the second intermediate attention feature map, can better adjust the attention of the input target feature map. Furthermore, by adding a residual connection structure to the target attention module, the acquired detailed information can be fully utilized, making the attribute recognition model have better discriminative power, thereby improving the model's attribute recognition accuracy and robustness, and meeting the trustworthiness characteristics.
[0028] In one possible implementation, the feature extraction process performed by the first feature extraction module includes:
[0029] The first feature extraction module is used to perform feature extraction on the intermediate feature map after it has been mapped by the last mapping module in the attribute recognition model.
[0030] In one possible implementation, the number of convolutional neural network structures included in the first feature extraction module of each mapping module is the same as the number of convolutional neural network structures included in the second feature extraction module; or
[0031] The convolutional neural network structures included in the first feature extraction module and the second feature extraction module in the attribute recognition model are deployed in a stacked manner, wherein the first feature extraction module includes the most convolutional neural network structures, and the second feature extraction module includes the fewest convolutional neural network structures.
[0032] The above method can deploy the number of convolutional neural network structures included in the feature extraction module in different ways, which improves the applicability of the model. In particular, when deploying in a stacked manner, the attribute recognition model can retain the detailed information required for attribute recognition, thereby improving the discriminative power of the attribute recognition model and thus improving the accuracy of attribute recognition.
[0033] In one possible implementation, the preprocessing of the input image containing the target object to obtain the image to be identified includes:
[0034] Using bilinear interpolation, the image size of the input image containing the target object is scaled to a preset size to obtain the image to be recognized.
[0035] Secondly, embodiments of this application provide a pedestrian attribute recognition device, comprising:
[0036] The preprocessing module is used to preprocess the input image containing the target object to obtain the image to be recognized;
[0037] An attribute prediction module is used to input the image to be identified into an attribute recognition model, and based on the attribute recognition model, to obtain the predicted probability that the target object has each reference attribute. The attribute recognition model is constructed based on a convolutional neural network, a target attention module, and a classifier. The target attention module is obtained by adding at least one global max pooling layer to a bilinear attention neural network structure.
[0038] The recognition result output module is used to obtain the attribute recognition result of the target object based on the predicted probability of each reference attribute.
[0039] In one possible implementation, the attribute prediction module is specifically used for:
[0040] Based on the attribute recognition model, the image to be recognized is sequentially processed by multiple mapping modules, processed by a first feature extraction module, and processed by the classifier through global average pooling, fully connected processing, and normalization to obtain the predicted probability of the target object having each reference attribute. Each of the multiple mapping modules is constructed based on the convolutional neural network and the target attention module, and the first feature extraction module is constructed based on the convolutional neural network.
[0041] In one possible implementation, any mapping module includes a second feature extraction module and the target attention module, wherein the second feature extraction module is constructed based on the convolutional neural network; the attribute prediction module is specifically used for:
[0042] Using the second feature extraction module, the target map is subjected to a first mapping process to obtain a target feature map;
[0043] Using the target attention module, the target feature map is subjected to a second mapping process to obtain an intermediate feature map output by any of the mapping modules;
[0044] Wherein, if any of the mapping modules is the first mapping module in the attribute recognition model, then the target image is the image to be recognized; if any of the mapping modules is not the first mapping module, then the target image is the intermediate feature map output by the preceding mapping module adjacent to any of the mapping modules.
[0045] In one possible implementation, the attribute prediction module is specifically used for:
[0046] Using the target attention module, perform the following steps:
[0047] The target feature map is subjected to dimensionality reduction and transformation processing to obtain the first feature map;
[0048] The first feature map is subjected to global max pooling and convolution to obtain a first intermediate attention feature map, and the first feature map is subjected to convolution to obtain a second feature map.
[0049] The first intermediate attention feature map and the second feature map are fused to obtain the second intermediate attention feature map, and the target attention feature map is obtained based on the target feature map and the second intermediate attention feature map.
[0050] The target feature map and the target attention feature map are subjected to residual processing to obtain an intermediate feature map output by any of the mapping modules.
[0051] In one possible implementation, the attribute prediction module is specifically used for:
[0052] The first feature extraction module is used to perform feature extraction on the intermediate feature map after it has been mapped by the last mapping module in the attribute recognition model.
[0053] In one possible implementation, the number of convolutional neural network structures included in the first feature extraction module of each mapping module is the same as the number of convolutional neural network structures included in the second feature extraction module; or
[0054] The convolutional neural network structures included in the first feature extraction module and the second feature extraction module in the attribute recognition model are deployed in a stacked manner, wherein the first feature extraction module includes the most convolutional neural network structures, and the second feature extraction module includes the fewest convolutional neural network structures.
[0055] In one possible implementation, the preprocessing module is specifically used for:
[0056] Using bilinear interpolation, the image size of the input image containing the target object is scaled to a preset size to obtain the image to be recognized.
[0057] Thirdly, embodiments of this application provide an electronic device, including:
[0058] Memory is used to store computer programs or instructions;
[0059] A processor for executing a computer program or instructions in the memory, such that the method described in any of the first aspects above is performed.
[0060] Fourthly, embodiments of this application provide a computer-readable storage medium that, when instructions in the storage medium are executed by a processor, enables the processor to perform the method described in any one of the first aspects above.
[0061] Furthermore, the technical effects of any of the implementation methods in the second to fourth aspects can be found in the technical effects of different implementation methods in the first aspect, and will not be repeated here. Attached Figure Description
[0062] Figure 1 This is a schematic diagram of an application scenario in the embodiments of this application;
[0063] Figure 2 This is a schematic diagram of another application scenario in the embodiments of this application;
[0064] Figure 3 This is a flowchart illustrating a pedestrian attribute recognition method according to an embodiment of this application;
[0065] Figure 4 This is a schematic diagram of the network architecture of an attribute recognition model in an embodiment of this application;
[0066] Figure 5A This is a schematic diagram of the network architecture of a mapping module in an embodiment of this application;
[0067] Figure 5B This is a schematic diagram of the network architecture of a second feature extraction module in an embodiment of this application;
[0068] Figure 5C This is a schematic diagram of the bilinear attention operation process of the target attention module in some preferred embodiments of this application;
[0069] Figure 5D This is a schematic diagram of the network architecture of a first feature extraction module in an embodiment of this application;
[0070] Figure 5E This is a schematic diagram of the operation flow of an Inception structure in an embodiment of this application;
[0071] Figure 5F This is a schematic diagram of the network architecture of a classifier in an embodiment of this application;
[0072] Figure 6 This is a schematic diagram of the specific network architecture of an attribute recognition model in an embodiment of this application;
[0073] Figure 7 This is a schematic diagram illustrating the specific process of a pedestrian attribute recognition method in an embodiment of this application;
[0074] Figure 8 This is a schematic diagram illustrating the process of mapping a target image using any mapping module in an embodiment of this application.
[0075] Figure 9 This is a schematic diagram of the second mapping process of a target feature map by a target attention module in an embodiment of this application;
[0076] Figure 10 This is a schematic diagram of the logical architecture of a pedestrian attribute recognition device in an embodiment of this application;
[0077] Figure 11 This is a schematic diagram of the physical architecture of the electronic device in the embodiments of this application. Detailed Implementation
[0078] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0079] It should be noted that the terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.
[0080] The design concept of the embodiments of this application will be briefly introduced below.
[0081] This application relates to the field of computer vision technology, and mainly to a method, device, equipment and storage medium for pedestrian attribute recognition.
[0082] With the development of video surveillance equipment and technology, more and more cameras are being deployed in every corner of the city to ensure public safety and social order. In surveillance applications, it is often necessary to analyze suspicious pedestrian targets in the video, and attribute recognition technology is an important method for pedestrian analysis.
[0083] Human Attribute Recognition (HAR) or Pedestrian Attributes Recognition (PAR) is the process of automatically identifying the attribute information of pedestrians from a given input image containing pedestrians using computer vision algorithms. This includes information such as gender, whether they wear glasses, age, hair length, whether they carry a backpack, clothing style, and the colors of their upper and lower body.
[0084] In the past, pedestrian attribute recognition relied on manual feature extraction and separate classifier training for each attribute. With the development of deep learning, more and more neural network structures have been proposed and applied to pedestrian attribute recognition tasks. Currently, most neural network models for pedestrian attribute recognition use classic convolutional neural network structures as their backbone, employing global average pooling (GAP) or average pooling (AVGPooling) to obtain global information from the input image. Then, based on this global information, the attribute information of pedestrians in the input image is obtained.
[0085] Due to limitations imposed by network structures such as GAP or AVGPooling, current models only acquire global information about the input image from attention maps obtained by transforming simplified feature representations. These simplified feature representations are calculated using first-order statistics such as mean, maximum, and minimum values from the feature maps. Therefore, the global information acquired by current neural network models does not include more refined features, such as hair and shoes, resulting in poor discriminative power and unstable attribute recognition accuracy.
[0086] In view of this, in order to solve the problem of unstable accuracy in traditional pedestrian attribute recognition methods, this application provides a pedestrian attribute recognition method. In this application embodiment, the input image containing the target object is preprocessed to obtain the image to be recognized; then the image to be recognized is input into the attribute recognition model, and based on the attribute recognition model, the predicted probability of the target object having each reference attribute is obtained. The attribute recognition model is constructed based on a convolutional neural network, a target attention module, and a classifier. The target attention module is obtained by adding at least one global max pooling layer to a bilinear attention neural network structure; then, based on the predicted probability of each reference attribute, the attribute recognition result of the target object in the image to be recognized is obtained.
[0087] Since the attribute recognition model in this embodiment includes a target attention module, compared with network structures such as GAP or AVGPooling in related technologies, the bilinear attention mechanism can capture second-order statistical information in the image, thereby enabling the attribute recognition model to obtain local nearest neighbor relationships in the image. In addition, the target attention module is obtained by adding at least one global max pooling layer to the bilinear attention neural network structure. Global max pooling can obtain the maximum value in the feature map of each channel, enabling the attribute recognition model to capture the most relevant elements in the image. This allows the target attention module to obtain more refined features, thereby giving the attribute recognition model better discriminative power. When performing attribute recognition tasks, it can improve the attribute recognition accuracy and robustness while maintaining the attribute recognition accuracy of the stable attribute recognition model, thus meeting the reliability characteristics.
[0088] The following describes, with reference to the accompanying drawings, an application scenario of an optional pedestrian attribute recognition method provided in this application embodiment.
[0089] like Figure 1As shown, this application scenario includes an electronic device 10. The electronic device 10 utilizes a pedestrian attribute recognition method provided in this application embodiment (the specific process will be described later) to preprocess an input image containing a target object to obtain an image to be recognized, and then performs attribute recognition on the target object in the image to obtain the attribute recognition result of the target object. The image can be any image from a preset image set.
[0090] In some feasible embodiments, the electronic device 10 can be various monitoring devices, smart terminals, smart mobile terminals, tablet computers, laptops, smart handheld devices, personal computers (PCs), computers, smart screens, various wearable devices, personal digital assistants (PDAs), smart home appliances, access control systems, vehicle terminals, aircraft, etc.
[0091] like Figure 2 As shown, this application scenario includes an electronic device 10 and a server 20. The electronic device 10 has an image acquisition function, and the electronic device 10 and the server 20 can communicate through a communication network.
[0092] Electronic device 10 is used to acquire an image containing a target object and send the image to server 20. After receiving the image sent by electronic device 10, server 20 uses a pedestrian attribute recognition method provided in this application embodiment (the specific process will be described later) to preprocess the acquired image containing the target object to obtain an image to be recognized, and performs attribute recognition on the target object in the image to be recognized to obtain the attribute recognition result of the target object.
[0093] In some feasible embodiments, server 20 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. Electronic device 10 and server 20 can be directly or indirectly connected via wired or wireless communication, and this application does not impose any limitations on this.
[0094] Of course, the methods provided in the embodiments of this application are not limited to... Figure 1 and Figure 2 The application scenarios shown can also be used in other possible application scenarios, and the embodiments of this application do not specifically limit them.
[0095] After introducing the application scenarios of the embodiments of this application, the preferred embodiments of this application will be further described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only for illustration and explanation of this application and are not intended to limit this application. Furthermore, the embodiments of this application and the features in the embodiments can be combined with each other without conflict.
[0096] See Figure 3 As shown in the figure, this application provides a method for pedestrian attribute recognition, and the specific process of the method is as follows:
[0097] Step 300: Preprocess the input image containing the target object to obtain the image to be recognized.
[0098] In this embodiment of the application, when performing step 300, bilinear interpolation can be used to scale the image size of the input image containing the target object to a preset size to obtain the image to be recognized.
[0099] Step 310: Input the image to be identified into the attribute recognition model, and based on the attribute recognition model, obtain the predicted probability of the target object having each reference attribute. The attribute recognition model is constructed based on a convolutional neural network, a target attention module, and a classifier. The target attention module is obtained by adding at least one global max pooling layer to the bilinear attention neural network structure.
[0100] In this embodiment of the application, before specifically implementing a pedestrian attribute recognition method, various reference attributes are preset, such as gender, age, whether glasses are worn, clothing style, etc. Then, a massive number of sample images are collected, and each sample image is labeled based on the preset reference attributes to obtain an image label for each sample image, such as... The label is an N-dimensional 0-1 vector, where 0 indicates that the sample image does not have the corresponding reference attribute, and 1 indicates that the sample image has the corresponding sample attribute. For example, if the reference attributes in a three-dimensional label vector are arranged as [whether male, whether short hair, whether wearing jeans], then the image label vector of a sample image containing a female, long hair and wearing jeans is [0, 0, 1].
[0101] The massive number of sample images collected, along with the image labels corresponding to each sample image, are used as training data. The attribute recognition model to be trained is then trained based on this training data. After the model reaches the convergence condition, the attribute recognition model after the last round of training is taken as the trained attribute recognition model. For details of the specific model training process, please refer to the subsequent procedures.
[0102] After obtaining the trained attribute recognition model, in the specific implementation, after obtaining the image to be recognized in step 300, when executing step 310, firstly, the image to be recognized is input into the pre-trained attribute recognition model. Then, based on the attribute recognition model, the image to be recognized is sequentially processed by multiple mapping modules, processed by the first feature extraction module, and processed by the classifier through global average pooling, fully connected processing, and normalization processing to obtain the predicted probability of the target object having each reference attribute.
[0103] Figure 4 This is a schematic diagram of the network architecture of an attribute recognition model in an embodiment of this application. (See also...) Figure 4 As shown, this attribute recognition model includes multiple mapping modules, a first feature extraction module, and a classifier. Each module will be described in detail below.
[0104] 1. Mapping Module
[0105] In this embodiment of the application, the attribute recognition model includes multiple mapping modules, wherein any one of the multiple mapping modules is constructed based on a convolutional neural network and a target attention module, and is used to perform mapping processing on the image or intermediate feature map of the input attribute recognition model.
[0106] Specifically, such as Figure 5A As shown, the mapping module includes a second feature extraction module and a target attention module, wherein,
[0107] The second feature extraction module is built on a convolutional neural network and is used to perform a first mapping process on the input target map to obtain a target feature map. If the mapping module is the first mapping module in the attribute recognition model, the target map is the image to be recognized input into the attribute recognition model. If the mapping module is not the first mapping module, the target map is the intermediate feature map output by the previous mapping module adjacent to the mapping module.
[0108] In some preferred embodiments, see Figure 5B As shown, the second feature extraction module includes a combination of multiple basic modules consisting of a classic convolutional neural network structure (Inception) + a batch normalization (BN) structure + a nonlinear activation transformation (ReLU) structure.
[0109] The target attention module is obtained by adding at least one global max pooling layer to the bilinear attention neural network structure. It is used to perform a second mapping process on the target feature map output by the second feature extraction module to obtain an intermediate feature map with more discriminative power. The main purpose of the second mapping process is to adjust the attention of the target feature map input to the target attention module in order to obtain more refined features, thereby making the attribute recognition model more discriminative, improving the accuracy and robustness of attribute recognition, and meeting the trustworthiness characteristics.
[0110] Figure 5C This is a schematic diagram illustrating the bilinear attention operation flow of the target attention module in some preferred embodiments of this application. (See also...) Figure 5C As shown, in some preferred embodiments of this application, network structures with dashed boxes 1 and 2 are added to the existing bilinear attention neural network structure. Specifically, dashed box 1 includes two branches and a dot product structure. The two branches include branch 1, a Global Max Pooling (GMP) module, and a convolution kernel size of [missing information]. convolutional structure Branch 2, convolution kernel size is convolutional structure The dashed frame 2 includes the residual connection structure.
[0111] For specific implementation details, please refer to... Figure 5C As shown, the specific execution flow of bilinear attention in the target attention module of any mapping module is as follows:
[0112] Suppose the feature map of the input target attention module is denoted as . , where c, h, and w represent the number of channels, height, and width of the feature map, respectively.
[0113] First, the target feature map is subjected to dimensionality reduction and transformation to obtain the first feature map.
[0114] In specific implementation, this feature map The size of a convolutional kernel input into the target attention module is convolutional structure Dimensionality reduction is performed to obtain a new feature map with reduced dimension along the channel direction. ,in, This represents the number of channels in the new feature map. Then, the dimensionality-reduced feature map... Divided along the channel direction A length of The feature vectors are then processed by performing an outer product on each feature vector to obtain a size. The matrix is denoted as Then, take the upper triangular portion of matrix V, and transform the matrix after taking the triangular portion into a matrix of length V. First feature map This can be expressed by the formula:
[0115]
[0116] Here, Vec() represents the operation of converting a matrix into a vector, and Utri() represents the operation of removing the upper triangular elements of a matrix. The above process can be simply represented as follows: ,like Figure 5C As shown.
[0117] Then, the obtained first feature map Global max pooling and convolution are performed to obtain the first intermediate attention feature map. , and the obtained first feature map Perform convolution processing to obtain the second feature map. .
[0118] In specific implementation, the obtained first feature map Inputting this into the GMP module branch yields a size of The feature map M is then input into a new convolutional kernel of size M. convolutional structure In the middle, we get a size of First intermediate attention map In this embodiment, the operation flow of the GMP module can be represented by the following formula:
[0119]
[0120] The above formula shows that the GMP module is used for extraction. Compared to the GAP method used in related technologies for feature extraction, the attribute recognition model in this embodiment can capture the most relevant elements in each target attention module, thereby achieving the goal of focusing on the first feature map. Attention adjustment is performed so that the first feature map is used as the basis for the adjustment. The resulting attention feature map includes more refined features.
[0121] At the same time, the obtained first feature map Input into a new convolutional structure with a kernel size of 1×1. Branching, resulting in the second feature map. That is, the input feature map is transformed through this convolution transformation. Feature maps converted to the original channel dimension .
[0122] Next, the first intermediate attention feature map and the second feature map are fused to obtain the second intermediate attention feature map Z, and then based on the target features... Figure X The target attention feature map Y is obtained by combining the second intermediate attention feature map Z with the second intermediate attention feature map Z.
[0123] In specific implementation, the first intermediate attention graph With the second feature map Perform channel multiplication, then pass the Sigmoid function to obtain a value of... Second intermediate attention feature map ; and the second intermediate attention feature map With the input target feature map Perform point-by-point multiplication to obtain the target attention feature map Y after attention map optimization.
[0124] To fully utilize the global and detailed information required for attribute recognition, this application embodiment adds a residual connection structure, such as... Figure 5C The dashed box 2 in the image represents the target features. Figure X The intermediate feature map is obtained by performing residual processing on the target attention feature map Y, resulting in the final output of the mapping module. It can be expressed by the following formula:
[0125] .
[0126] 2. First Feature Extraction Module
[0127] In this embodiment of the application, the attribute recognition model includes a first feature extraction module built on a convolutional neural network, which is used to perform feature extraction processing on the intermediate feature map output by the last mapping module in the attribute recognition model.
[0128] In some preferred embodiments, see Figure 5D As shown, the first feature extraction module includes a combination of multiple basic modules consisting of an Inception structure, a BN structure, and a ReLU structure.
[0129] In this embodiment, the operation flow of the Inception structure included in the first feature extraction module and the second feature extraction module can be similar.
[0130] In some preferred embodiments, see Figure 5E As shown, the Inception structure includes four branches and a Concat layer. The four branches include: Branch 1, Convolutional layer; branch 2, Convolutional layers Convolutional layer; branch 3, Convolutional layers Convolutional layers Convolutional layer; branch 4, Average pooling (AvgPool) layer Convolutional layer.
[0131] See also Figure 5E As shown, the operation flow of this Inception structure is as follows: Assume the feature map input to this Inception structure is denoted as... Then this feature Figure X The inputs are respectively fed into the convolution kernel with a size of 1. , and Of the three convolutional layer branches, and one In the average pooling branch, the output feature maps of the three convolutional branches and one pooling branch are then concatenated along the channel direction to obtain the feature map Y output by the Inception structure.
[0132] In this embodiment, BN in the first feature extraction module and the second feature extraction module represents a learnable normalization method used to prevent model overfitting; ReLU represents a nonlinear activation function used for nonlinear transformation, and its mathematical expression is: .
[0133] 3. Classifier
[0134] In this embodiment of the application, the attribute recognition model includes a classifier built on a global average pooling layer and a fully connected layer, which is used to classify attributes and output the predicted probability that the target object has each reference attribute.
[0135] In some preferred embodiments, see Figure 5F As shown, the classifier has a classification head structure, including a GAP layer, two fully connected layers, and a sigmoid layer; wherein,
[0136] The GAP layer is used to calculate the average value of the feature map output by the first feature extraction module according to each channel, so as to obtain a feature vector with dimension c of the number of channels.
[0137] Two fully connected layers are used together to map a feature vector of dimension c to a vector of dimension N of reference attributes;
[0138] The Sigmoid layer is used to map each element of a vector with dimension N equal to the number of reference attributes to the interval 0-1, which represents the probability value that the target object has the corresponding reference attribute.
[0139] In this embodiment, the operation flow of the GAP layer can be represented by the following formula:
[0140]
[0141] Among them, Y k X represents the feature vector of the k-th channel output by this GAP layer; k This represents the input feature map for this GAP layer; c, h, and w represent the number of channels, height, and width of the image input to this attribute recognition model, respectively; i and j represent the coordinates of the pixels.
[0142] In some feasible embodiments, the number of convolutional neural network structures included in the first feature extraction module in each mapping module can be the same as the number of convolutional neural network structures included in the second feature extraction module, wherein the convolutional neural network structure is the aforementioned basic module combination composed of Inception structure + BN structure + ReLU structure.
[0143] In other feasible embodiments, the number of convolutional neural network structures included in the first feature extraction module and the second feature extraction module in the attribute recognition model can also be deployed in a stacked manner. The first feature extraction module includes the most convolutional neural network structures, and the second feature extraction module includes the fewest. This convolutional neural network structure is the aforementioned basic module combination composed of Inception structure + BN structure + ReLU structure, such as... Figure 6 As shown, a progressive structure is adopted, which can fully preserve the detailed information needed for attribute recognition.
[0144] See Figure 6 As shown, the attribute recognition model includes two mapping modules, a first feature extraction module, and a classifier. The second feature extraction module in the first mapping module includes a combination of 9 basic modules consisting of Inception structure + BN structure + ReLU structure; the second feature extraction module in the second mapping module includes a combination of 6 basic modules consisting of Inception structure + BN structure + ReLU structure; and the first feature extraction module includes a combination of 3 basic modules consisting of Inception structure + BN structure + ReLU structure, which adopts a three-stage progressive decreasing style.
[0145] It should be noted that, Figure 5C , Figure 5E and Figure 6 The network architecture shown is merely an example and is not intended to be specific.
[0146] Step 320: Based on the predicted probabilities of each reference attribute, obtain the attribute recognition result of the target object.
[0147] In this embodiment of the application, when performing step 320, the predicted probability of each reference attribute is thresholded to obtain the attribute recognition result of the target object.
[0148] In specific implementation, when executing step 320, the predicted probability of each reference attribute is compared with a preset threshold. If the predicted probability of the reference attribute is greater than or equal to the preset threshold, the predicted probability of the reference attribute is replaced with a first standard value, indicating that the target object has the reference attribute. If the predicted probability of the reference attribute is less than the preset threshold, the predicted probability of the reference attribute is replaced with a second standard value, indicating that the target object does not have the reference attribute.
[0149] The following specific examples illustrate the training process of this attribute recognition model. However, it should be noted that this application does not limit the training method; any general training method can be used to complete the model training.
[0150] Still assuming the preset reference attributes are arranged as [whether male, whether short hair, whether wearing jeans], and based on each reference attribute, the image label of each sample image is obtained, where the sample image is a three-channel RGB image and the number of reference attributes N=3.
[0151] Then, based on each sample image and the image label of each sample image, the attribute recognition model to be trained is trained to obtain the model parameters θ of the trained attribute recognition model.
[0152] The following example, using a sample image I, illustrates the training process of the attribute recognition model to be trained. The network architecture of the attribute recognition model to be trained is as follows: Figure 6 As shown, the specific operations for training the attribute recognition model to be trained include, but are not limited to, the following:
[0153] Operation 1: Obtain sample image I and its corresponding image label. Among them, the label is A 0-1 dimension vector, where 0 indicates that the sample image does not have the corresponding reference attribute, and 1 indicates that the sample image has the corresponding reference attribute.
[0154] Operation 2: Preprocess the sample image I.
[0155] In practice, the sample image I is scaled to a uniform size using bilinear interpolation, such as... This refers to a uniformly sized image with a height of 256 pixels and a width of 196 pixels. During training, three data augmentation methods are simultaneously applied: random vertical flipping (e.g., with a probability of 0.5), random cropping (e.g., with a size of 10), and random edge padding, to enhance the robustness of the model; this can be expressed by the following formula:
[0156]
[0157] in, This represents a preprocessing mapping function that combines uniform scaling, random flipping, random cropping, and random padding.
[0158] Operation 3: Using the second feature extraction module in the first mapping module, process the original sample image. Perform the first mapping process to obtain the three-dimensional feature map. This refers to the aforementioned target feature map. This process can be represented by the following formula:
[0159]
[0160] in, express Figure 6 The network structure of the second feature extraction module in the first mapping module is shown, in order to This represents the structural parameters of the second feature extraction module within the first mapping module.
[0161] Operation 4: Process the feature map using the target attention module in the first mapping module. Perform a second mapping process to obtain the attention-corrected feature map. This refers to the target feature map output by the aforementioned mapping module. This process can also be represented by the following formula:
[0162]
[0163] in, express Figure 6 The network structure of the target attention module in the first mapping module is shown.
[0164] Operation 5: The intermediate feature map output by the first mapping module is mapped by the second feature extraction module and the target attention module in the second mapping module to obtain the intermediate feature map output by the second mapping module.
[0165] Operation 6: The first feature extraction module performs feature extraction processing on the intermediate feature map output by the second mapping module to obtain the final feature map. ,Right now
[0166]
[0167] in, This represents the structural parameters of the second feature extraction module within the second mapping module; This represents the structural parameters of the first feature extraction module.
[0168] Operation 7: Use a classifier to process the feature maps Mapped to an output vector of dimension N (number of reference attributes) .
[0169] Operation 8: Calculate the loss between the predicted value and the true value. If the loss value is greater than the loss threshold and the cumulative number of training rounds has not reached the round threshold, adjust the structural parameters in the attribute recognition model based on the loss value. If the loss value is not greater than the loss threshold and / or the cumulative number of training rounds has not reached the round threshold, end the training and output the attribute recognition model after the last round of training.
[0170] In this embodiment of the application, when training the attribute recognition model to be trained, the loss function can be a weighted Focal loss function, which can be expressed by the following formula:
[0171]
[0172] in, This represents the Sigmoid function, used to convert the output value into the range of 0-1; is a hyperparameter used to control the weights at the instance level, that is, based on the current predicted probability, it emphasizes the importance of hard-to-distinguish samples; This represents the prior probability distribution value of the c-th reference attribute.
[0173] During model training, AdamW is preferred as the optimization algorithm, where the algorithm parameters can be set to... This controls how the gradient is updated; the initial learning rate can be set to... The learning rate of the model can be adjusted using a cosine learning rate adjustment strategy; the weight decay rate can be set to 0.0005, the batch size can be set to 32, and the number of training rounds can be set to 90.
[0174] The following specific examples will be used to further explain the detailed process of the above pedestrian attribute recognition method.
[0175] Assume the input image containing the target object is a three-channel RGB image I.
[0176] See Figure 7 As shown, the specific implementation process of the pedestrian attribute recognition method in this application embodiment is as follows:
[0177] Step 700: Use bilinear interpolation to scale image I to obtain a scaled image I' of a preset size.
[0178] The preset size can be 256×196, but this application does not make any specific limitation.
[0179] Step 710: Input the scaled image I' into the attribute recognition model, and based on the attribute recognition model, obtain the predicted probability that the target object has each reference attribute.
[0180] In specific implementation, when executing step 710 based on the attribute recognition model to obtain the predicted probability of the target object having each reference attribute, the scaled image I' is processed sequentially through multiple mapping modules, through the first feature extraction module, and through a classifier for global average pooling, fully connected processing, and normalization processing, based on the attribute recognition model, to obtain the predicted probability of the target object in the scaled image I' having each reference attribute. Among these, any one of the multiple mapping modules is constructed based on a convolutional neural network and a target attention module, and the first feature extraction module is constructed based on a convolutional neural network.
[0181] In this application, any mapping module includes a second feature extraction module and a target attention module. The second feature extraction module is constructed based on the convolutional neural network. Therefore, when performing the mapping process on the scaled image I' through this mapping module, see [reference needed]. Figure 8 As shown, the specific steps are as follows:
[0182] Step 800: Using the second feature extraction module, perform the first mapping process on the target map to obtain the target feature map;
[0183] Step 810: Using the target attention module, perform a second mapping process on the target feature map to obtain the intermediate feature map output by the mapping module; wherein, if the mapping module is the first mapping module in the attribute recognition model, the target map is the scaled image I', and if the mapping module is not the first mapping module, the target map is the intermediate feature map output by the preceding mapping module adjacent to the mapping module.
[0184] In specific implementation, when performing step 810, refer to... Figure 9 As shown, the following steps are specifically performed using this target attention module:
[0185] Step 8101: Perform dimensionality reduction and transformation processing on the target feature map to obtain the first feature map;
[0186] Step 8102: Perform global max pooling and convolution on the first feature map to obtain the first intermediate attention feature map, and perform convolution on the first feature map to obtain the second feature map;
[0187] Step 8103: Fuse the first intermediate attention feature map and the second feature map to obtain the second intermediate attention feature map, and obtain the target attention feature map based on the target feature map and the second intermediate attention feature map;
[0188] Step 8104: Perform residual processing on the target feature map and the target attention feature map to obtain the intermediate feature map output by the mapping module.
[0189] In this embodiment of the application, the specific process for executing steps 8101 to 8104 is detailed below. Figure 5C and with Figure 5C The specific process of bilinear attention for the target attention module in any mapping module will not be elaborated here.
[0190] In this application, when performing the feature extraction process through the first feature extraction module, the first feature extraction module is specifically used to perform feature extraction processing on the intermediate feature map after it has been mapped by the last mapping module in the attribute recognition model.
[0191] Step 720: Threshold the predicted probabilities of each reference attribute to obtain the attribute recognition results of the target object.
[0192] In specific implementation, when executing step 720, the predicted probability of each reference attribute is compared with a preset threshold, and the attribute recognition result of the target object is obtained based on the comparison result.
[0193] Assume the preset threshold is 0.5. Then, when executing step 720, the predicted probability of each reference attribute is compared with the preset threshold (i.e., 0.5). If the predicted probability of the reference attribute is greater than or equal to 0.5, then the predicted probability of the reference attribute is replaced with 1 (i.e., the aforementioned first standard value), which means that the target object has the reference attribute; if the predicted probability of the reference attribute is less than 0.5, then the predicted probability of the reference attribute is replaced with 0 (i.e., the aforementioned second standard value), which means that the target object does not have the reference attribute.
[0194] Based on the same inventive concept, see [reference] Figure 10 As shown in the figure, this application provides a pedestrian attribute recognition device, including:
[0195] The preprocessing module 1010 is used to preprocess the input image containing the target object to obtain the image to be recognized;
[0196] The attribute prediction module 1020 is used to input the image to be identified into the attribute recognition model, and based on the attribute recognition model, obtain the predicted probability that the target object has each reference attribute. The attribute recognition model is constructed based on a convolutional neural network, a target attention module, and a classifier. The target attention module is obtained by adding at least one global max pooling layer to a bilinear attention neural network structure.
[0197] The recognition result output module 1030 is used to obtain the attribute recognition result of the target object based on the predicted probability of each reference attribute.
[0198] In one possible implementation, the attribute prediction module 1020 is specifically used for:
[0199] Based on the attribute recognition model, the image to be recognized is sequentially processed by multiple mapping modules, processed by a first feature extraction module, and processed by the classifier through global average pooling, fully connected processing, and normalization to obtain the predicted probability of the target object having each reference attribute. Each of the multiple mapping modules is constructed based on the convolutional neural network and the target attention module, and the first feature extraction module is constructed based on the convolutional neural network.
[0200] In one possible implementation, any mapping module includes a second feature extraction module and the target attention module, wherein the second feature extraction module is constructed based on the convolutional neural network; the attribute prediction module 1020 is specifically used for:
[0201] Using the second feature extraction module, the target map is subjected to a first mapping process to obtain a target feature map;
[0202] Using the target attention module, the target feature map is subjected to a second mapping process to obtain an intermediate feature map output by any of the mapping modules;
[0203] Wherein, if any of the mapping modules is the first mapping module in the attribute recognition model, then the target image is the image to be recognized; if any of the mapping modules is not the first mapping module, then the target image is the intermediate feature map output by the preceding mapping module adjacent to any of the mapping modules.
[0204] In one possible implementation, the attribute prediction module 1020 is specifically used for:
[0205] Using the target attention module, perform the following steps:
[0206] The target feature map is subjected to dimensionality reduction and transformation processing to obtain the first feature map;
[0207] The first feature map is subjected to global max pooling and convolution to obtain a first intermediate attention feature map, and the first feature map is subjected to convolution to obtain a second feature map.
[0208] The first intermediate attention feature map and the second feature map are fused to obtain the second intermediate attention feature map, and the target attention feature map is obtained based on the target feature map and the second intermediate attention feature map.
[0209] The target feature map and the target attention feature map are subjected to residual processing to obtain an intermediate feature map output by any of the mapping modules.
[0210] In one possible implementation, the attribute prediction module 1020 is specifically used for:
[0211] The first feature extraction module is used to perform feature extraction on the intermediate feature map after it has been mapped by the last mapping module in the attribute recognition model.
[0212] In one possible implementation, the number of convolutional neural network structures included in the first feature extraction module of each mapping module is the same as the number of convolutional neural network structures included in the second feature extraction module; or
[0213] The convolutional neural network structures included in the first feature extraction module and the second feature extraction module in the attribute recognition model are deployed in a stacked manner, wherein the first feature extraction module includes the most convolutional neural network structures, and the second feature extraction module includes the fewest convolutional neural network structures.
[0214] In one possible implementation, the preprocessing module 1010 is specifically used for:
[0215] Using bilinear interpolation, the image size of the input image containing the target object is scaled to a preset size to obtain the image to be recognized.
[0216] Based on the same inventive concept, see [reference] Figure 11 As shown in the embodiments of this application, an electronic device is provided, including:
[0217] Memory 111 is used to store computer programs or instructions;
[0218] Processor 112 is configured to execute computer programs or instructions in the memory such that any of the methods described in the above embodiments is performed.
[0219] The processor 112 may include one or more central processing units (CPUs) or digital processing units, etc., for executing computer programs or instructions in the memory 111, such that any of the methods described in the above embodiments is executed.
[0220] It should be noted that the specific connection medium between the memory 111 and the processor 112 is not limited in the embodiments of this application. The embodiments of this application... Figure 11 In this diagram, the memory 111 and processor 112 are connected via bus 113. The connections between other components are merely illustrative and not intended to be limiting. Bus 113 can be divided into address bus, data bus, control bus, etc. For ease of illustration, Figure 11 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0221] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium that, when executed by a processor, enables the processor to perform any of the methods described in the above embodiments. Since the principle by which the computer-readable storage medium solves the problem is similar to that of the pedestrian attribute recognition method, the implementation of the computer-readable storage medium can be found in the implementation of the method; repeated details will not be elaborated further.
[0222] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0223] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.
[0224] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flows in a flowchart and / or one or more blocks in a block diagram.
[0225] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.
[0226] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. A method for pedestrian attribute recognition, characterized in that, include: The input image containing the target object is preprocessed to obtain the image to be identified; The image to be identified is input into the attribute recognition model, and based on the attribute recognition model, the predicted probability of the target object having each reference attribute is obtained. The attribute recognition model is constructed based on a convolutional neural network, a target attention module, and a classifier. The target attention module is a module constructed by introducing a global max pooling operation into a bilinear attention neural network structure. It is used to simultaneously perform bilinear attention transformation and salient feature extraction based on global max pooling on the input feature map, and output an enhanced feature map through residual connections. Based on the predicted probabilities of each reference attribute, the attribute recognition result of the target object is obtained.
2. The method as described in claim 1, characterized in that, The step of obtaining the predicted probability that the target object possesses each reference attribute based on the attribute recognition model includes: Based on the attribute recognition model, the image to be recognized is sequentially processed by multiple mapping modules, processed by the first feature extraction module, and processed by the classifier through global average pooling, fully connected processing, and normalization to obtain the predicted probability of the target object having each reference attribute. Each of the plurality of mapping modules includes a second feature extraction module and the target attention module, wherein both the first feature extraction module and the second feature extraction module are constructed based on the convolutional neural network.
3. The method as described in claim 2, characterized in that, The mapping process, performed by any of the mapping modules, includes: Using the second feature extraction module, the target image is subjected to a first mapping process to obtain a target feature image; Using the target attention module, the target feature map is subjected to a second mapping process to obtain an intermediate feature map output by any of the mapping modules; Wherein, if any of the mapping modules is the first mapping module in the attribute recognition model, then the target image is the image to be recognized; if any of the mapping modules is not the first mapping module, then the target image is the intermediate feature map output by the preceding mapping module adjacent to any of the mapping modules.
4. The method as described in claim 3, characterized in that, The step of using the target attention module to perform a second mapping process on the target feature map to obtain an intermediate feature map output by any of the mapping modules includes: Using the target attention module, perform the following steps: The target feature map is subjected to dimensionality reduction and transformation processing to obtain the first feature map; The first feature map is subjected to global max pooling and convolution to obtain a first intermediate attention feature map, and the first feature map is subjected to convolution to obtain a second feature map. The first intermediate attention feature map and the second feature map are fused to obtain the second intermediate attention feature map, and the target attention feature map is obtained based on the target feature map and the second intermediate attention feature map. The target feature map and the target attention feature map are subjected to residual processing to obtain an intermediate feature map output by any of the mapping modules.
5. The method as described in claim 2, characterized in that, The feature extraction process performed by the first feature extraction module includes: The first feature extraction module is used to perform feature extraction on the intermediate feature map after it has been mapped by the last mapping module in the attribute recognition model.
6. The method as described in claim 3, characterized in that, The number of convolutional neural network structures included in the first feature extraction module of each mapping module is the same as the number of convolutional neural network structures included in the second feature extraction module; or The convolutional neural network structures included in the first feature extraction module and the second feature extraction module in the attribute recognition model are deployed in a stacked manner, wherein the first feature extraction module includes the most convolutional neural network structures, and the second feature extraction module includes the fewest convolutional neural network structures.
7. The method according to any one of claims 1-6, characterized in that, The preprocessing of the input image containing the target object to obtain the image to be identified includes: Using bilinear interpolation, the image size of the input image containing the target object is scaled to a preset size to obtain the image to be recognized.
8. A pedestrian attribute recognition device, characterized in that, include: The preprocessing module is used to preprocess the input image containing the target object to obtain the image to be recognized; An attribute prediction module is used to input the image to be identified into an attribute recognition model, and based on the attribute recognition model, obtain the predicted probability that the target object has each reference attribute. The attribute recognition model is constructed based on a convolutional neural network, a target attention module, and a classifier. The target attention module is a module constructed by introducing a global max pooling operation into a bilinear attention neural network structure. It is used to simultaneously perform bilinear attention transformation and salient feature extraction based on global max pooling on the input feature map, and output an enhanced feature map through residual connections. The recognition result output module is used to obtain the attribute recognition result of the target object based on the predicted probability of each reference attribute.
9. A display device, characterized in that, include: Memory is used to store computer programs or instructions; A processor for executing a computer program or instructions in the memory such that the method described in any one of claims 1-7 is performed.
10. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor, the processor is able to perform the method as described in any one of claims 1-7.