Attribute recognition method, system and device based on cross-domain auxiliary training and medium

Abstract

The application provides an attribute recognition method, system, device and medium based on cross-domain auxiliary training, which comprises the following steps: forming a human body reference image; performing human body part segmentation on each image in an image set without attribute labels to generate a human body part segmentation result, and randomly selecting a shielding picture; covering the shielding picture on the human body reference image to form a shielding synthesis image; judging whether each part of the human body in the human body reference image can be seen in the shielding synthesis image; if not, modifying the attribute label corresponding to the part, and forming the attribute label of the shielding synthesis image; extracting the features of each image and inputting them into a preset attribute prediction model and a preset human body main body prediction model to obtain attribute prediction results and human body main body prediction results. The application greatly improves the data quantity and diversity, realizes attribute recognition only on the main human body in the image, effectively and reasonably utilizes the synthesis data, and improves the stability and convergence speed of the application in the training stage.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology, and relates to a recognition method and system, particularly to an attribute recognition method, system, device and medium based on cross-domain assisted training. Background Technology

[0002] Pedestrian attributes are natural language descriptions of a pedestrian's physical characteristics. These typically include: biological attributes such as gender, age, and height; physical attributes such as clothing color and style; and accessories such as handbags, backpacks, and whether the pedestrian is wearing glasses. Pedestrian attribute recognition is a popular area of ​​computer vision technology with significant commercial application value. Currently, it has applications in video surveillance, pedestrian re-identification, facial recognition, and identity verification, among others.

[0003] Pedestrian attribute recognition is a multi-label classification problem. This research direction usually encounters the following problems: First, the image complexity is high. As the main body of the image, the human body may have various poses and various occlusions. At the same time, the imaging may have poor lighting and motion blur. Second, the data distribution is seriously unbalanced and has a long tail distribution. Third, due to the large workload of attribute labeling and the small amount of data, training is prone to overfitting.

[0004] Customer attribute recognition is a subcategory of pedestrian attribute recognition, involving the identification of customer attributes in shopping venues such as retail stores and supermarkets. This task, while sharing the challenges of traditional pedestrian attribute recognition, faces even greater difficulties due to the specific application scenarios, including more severe occlusion, such as occlusion between customers or between customers and shelves. Therefore, the amount of labeled data in this area is even smaller, making the challenge greater. Furthermore, identifying the main human figure in an image and recognizing only the attributes of that person when there is severe occlusion between customers is a significant challenge.

[0005] Therefore, how to provide an attribute recognition method, system, device, and medium based on cross-domain assisted training to solve the problems of occlusion and subject identification in customer attribute recognition that cannot be solved by existing technologies has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide an attribute recognition method, system, device and medium based on cross-domain assisted training, which can solve the problems of occlusion and subject identification in customer attribute recognition that cannot be solved by the prior art.

[0007] To achieve the above and other related objectives, this invention provides an attribute recognition method based on cross-domain assisted training, comprising: segmenting human body parts in an image with attribute labels to form a human body reference image; segmenting human body parts in each image in an image set without attribute labels to generate human body part segmentation results; randomly selecting an image from the human body part segmentation results as an occlusion image; overlaying the occlusion image onto the human body reference image to form an occlusion composite image, and determining whether each human body part obtained from the human body reference image is visible in the occlusion composite image; if so, no modification is needed to the attribute label corresponding to that part; if not, the attribute label corresponding to that part is modified accordingly; integrating the unmodified attribute label and the modified attribute label to form the attribute label of the occlusion composite image; extracting basic features from the image with attribute labels, the image set without attribute labels, and the occlusion composite image; inputting the basic features extracted from the image with attribute labels and the occlusion composite image into a preset attribute prediction model and a preset human body prediction model to obtain attribute prediction results and human body prediction results.

[0008] In one embodiment of the present invention, the attribute recognition method based on cross-domain assisted training further includes performing loss calculations on the attribute prediction results and the human subject prediction results respectively to obtain a preset attribute prediction model and a preset human subject prediction model corresponding to the minimum loss value, and using them as the attribute prediction application model and the human subject prediction application model.

[0009] In one embodiment of the present invention, in the actual application of the attribute recognition method based on cross-domain assisted training, the attribute recognition method based on cross-domain assisted training further includes: inputting an image to be predicted and extracting basic features from the image to be predicted; inputting the basic features extracted from the image to be predicted into the attribute prediction application model and the human body prediction application model to obtain the attributes of the image to be predicted and the human body in the image.

[0010] In one embodiment of the present invention, the step of overlaying the occlusion image onto the human body reference image to form an occlusion composite image, and determining whether various parts of the human body in the human body reference image are visible in the occlusion composite image includes: finding a head mask in the human body reference image, calculating the coordinates of the head center, randomly dividing the area from the head center coordinates to the bottom edge into N segments, and assigning a probability to each random segment that the matching occlusion point occurs in that segment.

[0011] In one embodiment of the present invention, the step of overlaying the occlusion image onto the human body reference image to form an occlusion composite image, and determining whether various parts of the human body in the human body reference image are visible in the occlusion composite image, further includes: calculating the coordinates of randomly generated occlusion points based on the head center coordinates and the probabilities assigned to N segments; the occlusion points are used to determine the anchor points on the human body reference image covered by the occlusion image; aligning the head center coordinates and the coordinates of the occlusion points, overlaying the occlusion image onto the human body reference image, and simultaneously removing non-human parts of the occlusion image according to the human body mask to form an occlusion composite image; calculating the occlusion ratio of each block in the human body reference image to determine whether various parts of the human body in the human body reference image are visible in the occlusion composite image; wherein, if the occlusion ratio of a part is greater than a preset ratio threshold, the part is determined to be invisible.

[0012] In one embodiment of the present invention, the step of calculating the coordinates of a randomly generated occlusion point includes: randomly obtaining a first random value between 0 and 1 in the x-direction, obtaining a second random value between 0 and 1 in the y-direction, and determining the segment to which the occlusion point belongs based on the second random value; obtaining a third random value between 0 and 1 in the y-direction based on the segment to which the occlusion point belongs; calculating the abscissa of the randomly generated occlusion point by multiplying the width of the occluded image by the first random value; and calculating the ordinate of the randomly generated occlusion point based on the ordinate of the head center coordinates, the distance from the head center to the top point of the segment, and the distance from the top point to the bottom point of the segment.

[0013] In one embodiment of the present invention, the step of inputting the basic features extracted from the image with attribute labels and the occluded composite image into a preset attribute prediction model and a preset human subject prediction model to obtain attribute prediction results and human subject prediction results includes: predicting the basic features extracted from the image with attribute labels and the occluded composite image using the preset attribute prediction model, and outputting several branches, each branch being responsible for predicting one attribute; wherein, the maximum index is searched in the branches to obtain the attribute prediction result; and predicting and drawing the basic features extracted from the occluded composite image and the image without attribute labels using the preset human subject prediction model, and outputting a prediction map, wherein the human subject prediction result includes pixels of human body regions as 1 and pixels of non-human body regions as 0 in the prediction map.

[0014] Another aspect of the present invention provides an attribute recognition system based on cross-domain assisted training, comprising: a segmentation module, used to segment human body parts in an image with attribute labels to form a human body reference image and to segment human body parts in each image in an image set without attribute labels to generate human body part segmentation results; an image synthesis and label generation module, used to randomly select an image from the human body part segmentation results as an occlusion image; to overlay the occlusion image onto the human body reference image to form an occlusion synthesis image, and to determine whether each human body part obtained from the human body reference image is visible in the occlusion synthesis image; if so, no modification is needed to the attribute label corresponding to that part; if not, the attribute label corresponding to that part is modified accordingly; to integrate the unmodified attribute label and the modified attribute label to form the attribute label of the occlusion synthesis image; a feature extraction module, used to extract basic features from the image with attribute labels, the image set without attribute labels, and the occlusion synthesis image; and a prediction module, used to input the basic features extracted from the image with attribute labels and the occlusion synthesis image into a preset attribute prediction model and a preset human body prediction model to obtain attribute prediction results and human body prediction results.

[0015] In another aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the attribute recognition method based on cross-domain assisted training.

[0016] The last aspect of the present invention provides an attribute recognition device based on cross-domain assisted training, comprising: a processor and a memory;

[0017] The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory, so that the attribute recognition device based on cross-domain assisted training executes the attribute recognition method based on cross-domain assisted training.

[0018] As described above, the attribute recognition method, system, device, and medium based on cross-domain assisted training of the present invention have the following beneficial effects:

[0019] First, the present invention proposes to simultaneously train with labeled synthetic data and unlabeled real data, using both types of data...

[0020] This method, which corresponds to different learning tasks, uses segmentation tasks to assist in category prediction training, significantly improving the quantity and diversity of data and enabling attribute recognition only on the main human subject in the image.

[0021] Second, the present invention proposes a method for correcting attribute labels in synthetic data by utilizing the correspondence between pedestrian attributes and human body part blocks. This effectively and rationally utilizes synthetic data, improving the stability and convergence speed of the invention during the training phase. Attached Figure Description

[0022] Figure 1 The diagram shown is a flowchart of an embodiment of the attribute recognition method based on cross-domain assisted training according to the present invention.

[0023] Figure 2 The diagram shown is a human body reference image of the present invention.

[0024] Figure 3 The diagram shown is an example of the present invention, which randomly divides the area from the center of the human head to the bottom edge into four segments.

[0025] Figure 4 The diagram shown is a schematic representation of the process of S13 of the present invention.

[0026] Figure 5 The diagram shown is a schematic of the preset human body prediction model and the preset attribute prediction model of the present invention.

[0027] Figure 6 The diagram shown is a schematic representation of the principle structure of the attribute recognition system based on cross-domain assisted training according to an embodiment of the present invention.

[0028] Component designation explanation

[0029] 6. Attributes based on cross-domain assisted training

[0030] Identification system

[0031] 61 Segmentation Module

[0032] 62 Image Synthesis and Label Generation Module

[0033] 63 Feature Extraction Module

[0034] 64 Prediction Module

[0035] 65 Loss Calculation Module

[0036] 66 Application Modules

[0037] 641 Main Body Segmentation Prediction Unit

[0038] 642 Attribute Prediction Units

[0039] Steps S11 to S18

[0040] Steps S131~S135 Detailed Implementation

[0041] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0042] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0043] This embodiment provides an attribute recognition method based on cross-domain assisted training, including:

[0044] Human body parts are segmented from images with attribute labels to form a baseline human body image;

[0045] Human body parts are segmented in each image in the image set without attribute labels to generate human body part segmentation results. One image is randomly selected from the human body part segmentation results as the occlusion image.

[0046] The occlusion image is overlaid on the human body reference image to form an occlusion composite image, and it is determined whether each part of the human body in the human body reference image is visible in the occlusion composite image; if so, there is no need to modify the attribute label corresponding to that part; if not, the attribute label corresponding to that part is modified accordingly; the unmodified attribute label and the modified attribute label are integrated to form the attribute label of the occlusion composite image.

[0047] Basic features are extracted from the images with attribute labels, the images in the image set without attribute labels, and the occluded composite images. The basic features extracted from the images with attribute labels and the occluded composite images are input into a preset attribute prediction model and a preset human subject prediction model to obtain attribute prediction results and human subject prediction results.

[0048] The following will describe in detail the attribute recognition method based on cross-domain assisted training provided in this embodiment, with reference to the accompanying diagrams. Please refer to... Figure 1 The diagram shows a flowchart of an attribute recognition method based on cross-domain assisted training in one embodiment. Figure 1 As shown, the attribute recognition method based on cross-domain assisted training specifically includes the following steps:

[0049] S11, segment human body parts from images with attribute labels to form a human baseline image.

[0050] In this embodiment, a pre-defined human body part segmentation network is used to segment the input image with attribute labels to obtain, for example... Figure 2 The human body reference image shown is an image stored in a two-dimensional array of the same size as the input image. The background value is 0, and the values ​​of the block pixels are filled from 1 to 14, that is, P1 is the head, P2 is the torso, P3 is the left upper arm, P4 is the left lower arm, P5 is the left palm, P6 is the right upper arm, P7 is the right lower arm, P8 is the right palm, P9 is the left thigh, P10 is the left calf, P11 is the left foot, P12 is the right thigh, P13 is the right calf, and P14 is the right foot.

[0051] S11 further includes segmenting each image in the image set without attribute labels into human body parts, generating human body part segmentation results, randomly selecting an image from the human body part segmentation results as an occlusion image, and scaling the occlusion image according to the human body proportions of the human body reference image.

[0052] S12: Locate the head mask in the human reference image, calculate the coordinates of the head center, and randomly divide the area from the head center coordinates to the bottom edge into N segments. Assign a probability to each random segment that a matching occlusion point occurs in that segment. In this embodiment, since the head mask is an irregular polygon, the head center coordinates C_add_head(x, y) can be obtained by calculating the center of this irregular polygon.

[0053] In this embodiment, the method for assigning a probability that a matching occlusion point occurs in a random segment includes:

[0054] Starting from the center coordinates of the head, the probability of using the first n / 4 segments is P1, P1 = 0.4 / n; the probability of using the remaining segments is P2, P2 = 1.2 / n. In this embodiment, it is necessary to calculate that P2 is 3 times P1, and the numerator of P1, 0.4, is an empirical value.

[0055] For example, such as Figure 3 The diagram shows four equal segments, with n = 4. The probability of using the segment from the center of the head to L1 is 0.4 / 4 = 0.1, and the probability of being assigned from L1 to L2, L2 to L3, and L3 to B is 1.2 / 4 = 0.3.

[0056] S13, the occlusion image is overlaid on the human body reference image to form an occlusion composite image, and it is determined whether each part of the human body in the human body reference image is visible in the occlusion composite image; if so, there is no need to modify the attribute label corresponding to the part; if not, the attribute label corresponding to the part is modified accordingly; the unmodified attribute label and the modified attribute label are integrated to form the attribute label of the occlusion composite image.

[0057] Please see Figure 4 The flowchart shown is for S13. Figure 4 As shown, S14 includes:

[0058] S131, calculate the coordinates C_occ(x, y) of the randomly generated occlusion points based on the coordinates of the human head center and the probabilities assigned to the N segments; the occlusion points are used to determine the anchor points on the human body reference image covered by the occlusion image.

[0059] The steps for calculating the coordinates of randomly generated occlusion points include:

[0060] A first random value Px between 0 and 1 is randomly obtained in the x-direction, and a second random value between 0 and 1 is obtained in the y-direction. The segment to which the occluded point belongs is determined based on the second random value.

[0061] For example, a second random value between 0 and 1, 0.5, is obtained in the y-direction, and the randomly generated occlusion point is located as follows: Figure 3 Between L2 and L3 as shown.

[0062] Based on the segment to which the occlusion point belongs, a third random value Py between 0 and 1 is obtained in the y direction;

[0063] The x-coordinate of the randomly generated occlusion point is calculated by multiplying the width of the occluded image by the first random value.

[0064] Specifically, the x-coordinate of the randomly generated occlusion point is x = width * Px.

[0065] Based on the ordinate of the head center, the distance from the head center to the top point of the segment, and the distance from the top point of the segment to the bottom point, calculate the ordinate of the randomly generated occlusion point.

[0066] Specifically, the ordinate of the randomly generated occlusion point is y = c_y + dist(c_y, L2) + dist(L2, L3) * Py.

[0067] Where c_y is the y-coordinate of the center of the head in the figure, dist(c_y,L2) is the distance from the center of the head to L2, and dist(L2,L3) is the distance from L2 to L3.

[0068] S132, Align the coordinates of the head center and the coordinates of the occlusion point, overlay the occlusion image onto the human body reference image, and simultaneously remove non-human parts of the occlusion image according to the human body mask to form an occlusion composite image.

[0069] In this embodiment, since the pixel value of each pixel in the human body block mask image represents what block that pixel belongs to (background, head, torso, arm, etc.), the criterion for elimination is to remove pixels of non-human body parts (background).

[0070] S133, calculate the occlusion ratio of each block in the human body reference image that is occluded by the occluded image, so as to determine whether each part of the human body in the human body reference image obtained in the occluded composite image is visible. If the occlusion ratio of a part is greater than a preset ratio threshold, then it is determined that the part is not visible.

[0071] In this embodiment, the occlusion ratio of each block in the human body reference image is equal to the area of ​​the block before occlusion divided by the remaining area of ​​the block after occlusion.

[0072] In this embodiment, if the occlusion ratio of a part is greater than a preset ratio threshold, then the part is determined to be invisible.

[0073] S134 If the part is determined to be visible, there is no need to modify the attribute label corresponding to the part, that is, the attributes of the image with the attribute label are maintained.

[0074] S135, if it is determined that the part is not visible, the corresponding attribute label of the part is modified accordingly; the unmodified attribute label and the modified attribute label are integrated to form the attribute label of the occlusion composite image.

[0075] S14, perform basic feature extraction on the images with attribute labels, the images in the image set without attribute labels, and the occluded composite images.

[0076] Specifically, this embodiment uses a pre-stored convolutional neural network to calculate high-dimensional feature maps for each image. The weights of this convolutional neural network are pre-trained in other training tasks, such as using a ResNet-101 convolutional neural network.

[0077] S15, the basic features extracted from the image with attribute labels and the occluded composite image are input into the preset attribute prediction model and the preset human subject prediction model to obtain the attribute prediction results and the human subject prediction results.

[0078] Please see Figure 5 The image shows a schematic diagram of the preset human body prediction model and the preset attribute prediction model.

[0079] In this embodiment, S15 includes:

[0080] After predicting and drawing the basic features extracted from the occluded synthetic image and the image set without attribute labels by using a preset human subject prediction model, a prediction map is output. The human subject prediction result includes a human body region pixel as 1 and a non-human body region pixel as 0 in the prediction map.

[0081] like Figure 5 As shown, firstly, the occluded composite image and the image without attribute labels enter the N-layer Bottleneck layer. In this embodiment, N is set to 3. Then, it enters a feature transformation layer to normalize the output dimension to 64 dimensions. The resulting feature map will then be output to the second layer of the preset attribute prediction model, and simultaneously enter a convolutional layer with an output dimension of 1 to obtain a prediction map.

[0082] After predicting the basic features extracted from the image with attribute labels and the occluded composite image by the preset attribute prediction model, several branches are output, each branch is responsible for predicting one attribute; wherein, the maximum index is searched in the branches to obtain the attribute prediction result.

[0083] like Figure 5 As shown, firstly, the image with attribute labels and the synthesized occluded image enter an N-layer Bottleneck layer. In this embodiment, N is set to 3. Then, it enters a feature transformation layer to normalize the output dimension to 64 dimensions. The feature map of the second layer is concatenated with the feature map output from the second layer of the human body segmentation prediction branch. After passing through a 1x1 convolutional kernel and a convolutional layer with an output dimension of 64, it is divided into 10 branches, each responsible for predicting a single attribute.

[0084] S16, perform loss calculations on the attribute prediction results and the human body prediction results respectively to obtain the preset attribute prediction model and the preset human body prediction model corresponding to the minimum loss value, and use them as the attribute prediction application model and the human body prediction application model.

[0085] In this embodiment, cross-entropy loss Loss1 is used to calculate the loss of the attribute prediction results.

[0086] Specifically, the formula for calculating cross-entropy loss is:

[0087]

[0088] Where K is the total number of categories. For example, the value is 61 for the age category, 3 for the gender category, 11 for the top, bottom, and shoe colors categories, 3 for the top types, 4 for the bottom types, and 3 for the hats, glasses, and backpacks. cIt is a one-hot encoding, corresponding to a category of 1, otherwise it is 0. p c These are the predicted values.

[0089] The attribute prediction formula is: Where N is the number of attributes, for example, a value of 10.

[0090] In this embodiment, Loss2 is used to calculate the loss of the predicted human body.

[0091] The formula for calculating the loss of the human body prediction result is as follows: Where n is the total number of feature map points, y i p is the truth value at a certain point. i It is the predicted value at a certain point.

[0092] The final loss is: Loss = m1(Loss1 + Loss2) + m2Loss2. Where m1 is the index mask of the attribute-labeled training data, with a mask value of 1; m2 is also the index mask of the attribute-labeled training data, with a mask value of 1. The goal is to use attribute-labeled training data (including real and synthetic data) for attribute prediction and human body segmentation prediction learning, and to use unlabeled training data for human body segmentation prediction learning.

[0093] S17, in the actual application of the attribute recognition method based on cross-domain assisted training, the image to be predicted is input, and then the process proceeds to S14 to extract basic features from the image to be predicted, and S15 to input the basic features extracted from the image to be predicted into the attribute prediction application model and the human subject prediction application model, so as to obtain the attributes of the image to be predicted and the human subject in the image.

[0094] The attribute recognition method based on cross-domain assisted training described in this embodiment has the following beneficial effects:

[0095] First, the attribute recognition method based on cross-domain assisted training described in this embodiment proposes a method that simultaneously uses labeled synthetic data and unlabeled real data for training, with the two types of data corresponding to different learning tasks, wherein the segmentation task is used to assist the training of the category prediction task. This greatly improves the quantity and diversity of data, and enables attribute recognition only on the main human body in the image.

[0096] Second, the attribute recognition method based on cross-domain assisted training described in this embodiment proposes a method to correct the attribute labeling of synthetic data by utilizing the correspondence between pedestrian attributes and human body part blocks. This effectively and rationally utilizes synthetic data, improving the stability and convergence speed of the invention during the training phase.

[0097] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following... Figure 1 The attribute recognition method based on cross-domain assisted training.

[0098] At any possible level of technical detail, this application can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this application.

[0099] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0100] The computer-readable program described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and forwards these instructions to a computer-readable storage medium in the respective computing / processing device. The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and procedural programming languages ​​such as the "C" language or similar programming languages. Computer-readable program instructions may execute entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can execute computer-readable program instructions to implement various aspects of this application by utilizing state information from the computer-readable program instructions.

[0101] This embodiment also provides an attribute recognition system based on cross-domain assisted training, including:

[0102] The segmentation module is used to segment human body parts in images with attribute labels to form a human body baseline image, and to segment human body parts in each image of images without attribute labels to generate human body part segmentation results.

[0103] The image synthesis and label generation module is used to randomly select an image from the human body part segmentation results as an occlusion image; overlay the occlusion image onto the human body reference image to form an occlusion synthesis image, and determine whether each part of the human body in the human body reference image is visible in the occlusion synthesis image; if so, there is no need to modify the attribute label corresponding to that part; if not, the attribute label corresponding to that part is modified accordingly; and integrate the unmodified attribute label and the modified attribute label to form the attribute label of the occlusion synthesis image.

[0104] The feature extraction module is used to extract basic features from the images with attribute labels, the images in the image set without attribute labels, and the occluded composite images;

[0105] The prediction module is used to input the basic features extracted from the image with attribute labels and the occluded synthetic image into a preset attribute prediction model and a preset human subject prediction model to obtain attribute prediction results and human subject prediction results.

[0106] The following will describe in detail the attribute recognition system based on cross-domain assisted training provided in this embodiment, with reference to the accompanying diagrams. Please refer to... Figure 6 The diagram shows the principle structure of an attribute recognition system based on cross-domain assisted training in one embodiment. Figure 6 As shown, the attribute recognition system 6 based on cross-domain assisted training includes: a segmentation module 61, an image synthesis and label generation module 62, a feature extraction module 63, a prediction module 64, and a loss calculation module 65. The prediction module 64 includes a human body segmentation prediction unit 641 and an attribute prediction unit 642.

[0107] The segmentation module 61 is used to segment human body parts in an image with attribute labels to form a human body reference image.

[0108] In this embodiment, the segmentation module 61 segments the input image with attribute labels using a preset human body part segmentation network to obtain, for example... Figure 2 The human body reference image shown is an image stored in a two-dimensional array of the same size as the input image. The background value is 0, and the values ​​of the block pixels are filled from 1 to 14, that is, P1 is the head, P2 is the torso, P3 is the left upper arm, P4 is the left lower arm, P5 is the left palm, P6 is the right upper arm, P7 is the right lower arm, P8 is the right palm, P9 is the left thigh, P10 is the left calf, P11 is the left foot, P12 is the right thigh, P13 is the right calf, and P14 is the right foot.

[0109] The segmentation module 61 is further configured to segment human body parts in each image in the image set without attribute labels, generate human body part segmentation results, randomly select an image from the human body part segmentation results as an occlusion image, and scale the occlusion image according to the human body proportion of the human body reference image.

[0110] The image synthesis and label generation module 62 is used to find the head mask in the human reference image, calculate the coordinates of the head center, and randomly divide the area from the head center coordinates to the bottom edge into N segments, assigning a probability to each random segment that a matching occlusion point occurs in that segment. In this embodiment, since the head mask is an irregular polygon, the head center coordinates C_add_head(x, y) can be obtained by calculating the center of this irregular polygon.

[0111] In this embodiment, the process by which the image synthesis and label generation module 62 assigns the probability of a matching occlusion point occurring in a random segment includes: starting from the center coordinates of the human head, the first n / 4 segments use probability P1, P1 = 0.4 / n; and the remaining segments use probability P2, P2 = 1.2 / n.

[0112] The image synthesis and label generation module 62 is further configured to overlay the occluded image onto the human body reference image to form an occluded composite image, and determine whether each part of the human body in the human body reference image is visible in the occluded composite image; if so, there is no need to modify the attribute label corresponding to that part; if not, the attribute label corresponding to that part is modified accordingly; the unmodified attribute label and the modified attribute label are integrated to form the attribute label of the occluded composite image.

[0113] The image synthesis and label generation module 62 calculates the coordinates C_occ(x, y) of randomly generated occlusion points based on the head center coordinates and the probabilities assigned to N segments; aligns the head center coordinates with the occlusion point coordinates, overlays the occlusion image onto the human body reference image, and simultaneously removes non-human parts of the occlusion image according to the human body mask to form an occlusion synthesis image; calculates the occlusion ratio of each block in the human body reference image to determine whether each part of the human body in the human body reference image is visible in the occlusion synthesis image; if the occlusion ratio of a part is greater than a preset ratio threshold, the part is determined to be invisible; if the part is determined to be visible, there is no need to modify the attribute label corresponding to the part, i.e., the attributes of the image with attribute labels are maintained; if the part is determined to be invisible, the attribute label corresponding to the part is modified accordingly; integrates the unmodified attribute labels and the modified attribute labels to form the attribute labels of the occlusion synthesis image.

[0114] The occlusion point is used to determine the anchor point on the human body reference image covered by the occlusion image.

[0115] In this embodiment, the process by which the image synthesis and label generation module 62 calculates the coordinates of randomly generated occlusion points includes:

[0116] A first random value Px between 0 and 1 is randomly obtained in the x-direction, and a second random value between 0 and 1 is obtained in the y-direction. The segment to which the occluded point belongs is determined based on the second random value.

[0117] For example, a second random value between 0 and 1, 0.5, is obtained in the y-direction, and the randomly generated occlusion point is located as follows: Figure 3 Between L2 and L3 as shown.

[0118] Based on the segment to which the occlusion point belongs, a third random value Py between 0 and 1 is obtained in the y direction;

[0119] The x-coordinate of the randomly generated occlusion point is calculated by multiplying the width of the occluded image by the first random value.

[0120] Specifically, the x-coordinate of the randomly generated occlusion point is x = width * Px.

[0121] Based on the ordinate of the head center, the distance from the head center to the top point of the segment, and the distance from the top point of the segment to the bottom point, calculate the ordinate of the randomly generated occlusion point.

[0122] Specifically, the ordinate of the randomly generated occlusion point is y = c_y + dist(c_y, L2) + dist(L2, L3) * Py.

[0123] Where c_y is the y-coordinate of the center of the head in the figure, dist(c_y,L2) is the distance from the center of the head to L2, and dist(L2,L3) is the distance from L2 to L3.

[0124] In this embodiment, the occlusion ratio of each block in the human body reference image is equal to the area of ​​the block before occlusion divided by the remaining area of ​​the block after occlusion. In this embodiment, if the occlusion ratio of a part is greater than a preset ratio threshold, then that part is determined to be invisible.

[0125] The feature extraction module 63 performs basic feature extraction on the images with attribute labels, the images in the image set without attribute labels, and the occluded composite images.

[0126] Specifically, the feature extraction module 63 uses a pre-stored convolutional neural network to calculate high-dimensional feature maps for each image. The weights of this convolutional neural network are pre-trained in other training tasks, such as using a ResNet-101 convolutional neural network.

[0127] The prediction module 64 is used to input the basic features extracted from the image with attribute labels and the occluded composite image into a preset attribute prediction model and a preset human subject prediction model to obtain attribute prediction results and human subject prediction results. The prediction module 64 includes a human subject segmentation prediction unit 641 and an attribute prediction unit 642.

[0128] The main human body segmentation prediction unit 641 predicts and draws the basic features extracted from the occluded synthetic image and the image set without attribute labels by using a preset human body prediction model, and outputs a prediction map. The human body prediction result includes a prediction map in which the pixels of the human body area are 1 and the pixels of the non-human body area are 0.

[0129] The main body segmentation prediction unit 641 first inputs the occluded synthetic image and the image without attribute labels into an N-layer Bottleneck layer. In this embodiment, N is 3. Then, it inputs a feature transformation layer to normalize the output dimension to 64 dimensions. The resulting feature map is then output to the second layer of the preset attribute prediction model and simultaneously inputs a convolutional layer with an output dimension of 1 to obtain a prediction map.

[0130] The attribute prediction unit 642 predicts the basic features extracted from the image with attribute labels and the occluded composite image using the preset attribute prediction model, and outputs several branches, each responsible for predicting one attribute; wherein, the maximum index is searched in the branches to obtain the attribute prediction result.

[0131] The attribute prediction unit 642 first processes the image with attribute labels and the synthesized occlusion image into an N-layer Bottleneck layer (N is 3 in this embodiment). Then, it enters a feature transformation layer to normalize the output dimension to 64 dimensions. The feature map from the second layer is then concatenated with the feature map output from the second layer of the body segmentation prediction branch. After passing through a 1x1 convolutional kernel with an output dimension of 64, it is divided into 10 branches, each responsible for predicting a single attribute.

[0132] The loss calculation module 65 is used to perform loss calculations on the attribute prediction results and the human body prediction results respectively, so as to obtain the preset attribute prediction model and the preset human body prediction model corresponding to the minimum loss value, and use them as the attribute prediction application model and the human body prediction application model.

[0133] The application module 66 is used to receive the image to be predicted in the actual application of the attribute recognition system based on cross-domain assisted training, and input the image to be predicted into the feature extraction module 63 to extract basic features from the image to be predicted, and input the basic features extracted from the image to be predicted into the attribute prediction application model and the human subject prediction application model through the prediction module 64 to obtain the attributes of the image to be predicted and the human subject in the image.

[0134] It should be noted that the division of the various modules in the above system is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls, entirely in hardware, or partially in software calls via processing element calls, with some modules implemented in hardware. For example, module x can be a separate processing element or integrated into a chip within the system. Additionally, module x can be stored as program code in the system's memory, invoked and executed by a processing element. The implementation of other modules is similar. These modules can be fully or partially integrated together or implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the processor element or through software instructions. These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), one or more Field Programmable Gate Arrays (FPGAs), etc. When a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. These modules can be integrated together to form a System-on-a-Chip (SOC).

[0135] Example 2

[0136] This application provides an attribute recognition system based on cross-domain assisted training. The attribute recognition system based on cross-domain assisted training includes: a processor, a memory, a transceiver, a communication interface and / or a system bus; the memory and the communication interface are connected to the processor and the transceiver through the system bus and complete mutual communication; the memory is used to store computer programs; the communication interface is used to communicate with other devices; the processor and the transceiver are used to run the computer programs, so that the attribute recognition system based on cross-domain assisted training executes the various steps of the attribute recognition method based on cross-domain assisted training as described in Embodiment 1.

[0137] The system bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices (such as clients, read-write libraries, and read-only libraries). Memory may include Random Access Memory (RAM) and may also include non-volatile memory, such as at least one disk drive.

[0138] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0139] The scope of protection of the attribute recognition method based on cross-domain assisted training described in this invention is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this invention is included within the scope of protection of this invention.

[0140] The present invention also provides an attribute recognition system based on cross-domain assisted training. The attribute recognition system based on cross-domain assisted training can implement the attribute recognition method based on cross-domain assisted training described in the present invention. However, the implementation device of the attribute recognition method based on cross-domain assisted training described in the present invention includes, but is not limited to, the structure of the attribute recognition system based on cross-domain assisted training listed in this embodiment. All structural modifications and substitutions of the prior art made in accordance with the principles of the present invention are included within the protection scope of the present invention.

[0141] In summary, the attribute recognition method, system, device, and medium based on cross-domain assisted training described in this invention have the following beneficial effects:

[0142] First, the present invention proposes a method that simultaneously trains using labeled synthetic data and unlabeled real data, with the two types of data corresponding to different learning tasks, wherein the segmentation task is used to assist in the training of the category prediction task. This significantly improves the quantity and diversity of data and enables attribute recognition only on the main human subject in the image.

[0143] Secondly, the method proposed in this invention utilizes the correspondence between pedestrian attributes and human body part blocks to correct the attribute labels of synthetic data. This effectively and rationally utilizes synthetic data, improving the stability and convergence speed of the invention during the training phase. Therefore, this invention effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0144] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. An attribute recognition method based on cross-domain assisted training, characterized in that, include: Human body parts are segmented from images with attribute labels to form a baseline human body image; Human body parts are segmented in each image in the image set without attribute labels to generate human body part segmentation results. One image is randomly selected from the human body part segmentation results as the occlusion image. The occlusion image is overlaid on the human body reference image to form an occlusion composite image, and it is determined whether each part of the human body in the human body reference image is visible in the occlusion composite image; if so, there is no need to modify the attribute label corresponding to that part; if not, the attribute label corresponding to that part is modified accordingly; the unmodified attribute label and the modified attribute label are integrated to form the attribute label of the occlusion composite image. Find the head mask in the human reference image, calculate the coordinates of the head center, and randomly divide the area from the head center coordinates to the bottom edge into N segments, and assign the probability of the matching occlusion point occurring in the segment to the random segment. Based on the coordinates of the human head center and the probability assigned to N segments, the coordinates of randomly generated occlusion points are calculated; these occlusion points are used to determine the anchor points on the human body reference image covered by the occlusion image. Align the coordinates of the center of the human head with the coordinates of the occlusion point, overlay the occlusion image onto the human body reference image, and simultaneously remove non-human parts of the occlusion image according to the human body mask to form an occlusion composite image. The occlusion ratio of each block in the human body reference image is calculated to determine whether each part of the human body in the human body reference image is visible in the occluded composite image; wherein, if the occlusion ratio of a part is greater than a preset ratio threshold, the part is determined to be invisible. The steps for calculating the coordinates of randomly generated occlusion points include: A first random value between 0 and 1 is randomly obtained in the x-direction, and a second random value between 0 and 1 is obtained in the y-direction. The segment to which the occlusion point belongs is determined based on the position of the second random value in the N segments. Based on the segment to which the occlusion point belongs, a third random value between 0 and 1 is obtained in the y direction; The x-coordinate of the randomly generated occlusion point is calculated by multiplying the width of the occluded image by the first random value. Based on the ordinate of the head center coordinate, the distance from the head center to the top point of the segment, the distance from the top point of the segment to the bottom point, and the third random value, calculate the ordinate of the randomly generated occlusion point. Basic features are extracted from the images with attribute labels, the images in the image set without attribute labels, and the occluded composite images. The basic features extracted from the images with attribute labels and the occluded composite images are input into a preset attribute prediction model and a preset human subject prediction model to obtain attribute prediction results and human subject prediction results. After predicting the basic features extracted from the image with attribute labels and the occluded synthetic image by the preset attribute prediction model, several branches are output, each branch is responsible for predicting one attribute; wherein, the maximum index is searched in the branches to obtain the attribute prediction result; After predicting and drawing the basic features extracted from the occluded synthetic image and the image set without attribute labels by using a preset human subject prediction model, a prediction map is output. The human subject prediction result includes a human body region pixel as 1 and a non-human body region pixel as 0 in the prediction map.

2. The attribute recognition method based on cross-domain assisted training according to claim 1, characterized in that, The attribute recognition method based on cross-domain assisted training further includes performing loss calculations on the attribute prediction results and the human subject prediction results respectively to obtain the preset attribute prediction model and the preset human subject prediction model corresponding to the minimum loss value, and using them as the attribute prediction application model and the human subject prediction application model.

3. The attribute recognition method based on cross-domain assisted training according to claim 2, characterized in that, In the practical application of the attribute recognition method based on cross-domain assisted training, the method further includes: Input the image to be predicted and extract its basic features. The basic features extracted from the image to be predicted are input into the attribute prediction application model and the human subject prediction application model to obtain the attributes of the image to be predicted and the human subject in the image.

4. An attribute recognition system based on cross-domain assisted training, characterized in that, include: The segmentation module is used to segment human body parts in images with attribute labels to form a human body baseline image, and to segment human body parts in each image of images without attribute labels to generate human body part segmentation results. An image synthesis and label generation module is used to randomly select an image from the human body part segmentation results as an occlusion image; The occlusion image is overlaid on the human body reference image to form an occlusion composite image, and it is determined whether each part of the human body in the human body reference image is visible in the occlusion composite image; if so, there is no need to modify the attribute label corresponding to that part; if not, the attribute label corresponding to that part is modified accordingly; the unmodified attribute label and the modified attribute label are integrated to form the attribute label of the occlusion composite image. Find the head mask in the human reference image, calculate the coordinates of the head center, and randomly divide the area from the head center coordinates to the bottom edge into N segments, and assign the probability of the matching occlusion point occurring in the segment to the random segment. Based on the coordinates of the human head center and the probability assigned to N segments, the coordinates of randomly generated occlusion points are calculated; these occlusion points are used to determine the anchor points on the human body reference image covered by the occlusion image. Align the coordinates of the center of the human head with the coordinates of the occlusion point, overlay the occlusion image onto the human body reference image, and simultaneously remove non-human parts of the occlusion image according to the human body mask to form an occlusion composite image. The occlusion ratio of each block in the human body reference image is calculated to determine whether each part of the human body in the human body reference image is visible in the occluded composite image; wherein, if the occlusion ratio of a part is greater than a preset ratio threshold, the part is determined to be invisible. The steps for calculating the coordinates of randomly generated occlusion points include: A first random value between 0 and 1 is randomly obtained in the x-direction, and a second random value between 0 and 1 is obtained in the y-direction. The segment to which the occlusion point belongs is determined based on the position of the second random value in the N segments. Based on the segment to which the occlusion point belongs, a third random value between 0 and 1 is obtained in the y direction; The x-coordinate of the randomly generated occlusion point is calculated by multiplying the width of the occluded image by the first random value. Based on the ordinate of the head center coordinate, the distance from the head center to the top point of the segment, the distance from the top point of the segment to the bottom point, and the third random value, calculate the ordinate of the randomly generated occlusion point. The feature extraction module is used to extract basic features from the images with attribute labels, the images in the image set without attribute labels, and the occluded composite images; The prediction module is used to input the basic features extracted from the image with attribute labels and the occluded composite image into a preset attribute prediction model and a preset human subject prediction model to obtain attribute prediction results and human subject prediction results; after predicting the basic features extracted from the image with attribute labels and the occluded composite image through the preset attribute prediction model, it outputs several branches, each branch is responsible for predicting one attribute; wherein, the maximum index is searched in the branches to obtain the attribute prediction result; after predicting and drawing the basic features extracted from the occluded composite image and the image without attribute labels through the preset human subject prediction model, it outputs a prediction map, wherein the human subject prediction result includes pixels of human body area as 1 and pixels of non-human body area as 0 in the prediction map.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the attribute recognition method based on cross-domain assisted training as described in any one of claims 1 to 3.

6. An attribute recognition device based on cross-domain assisted training, characterized in that, include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the attribute recognition device based on cross-domain assisted training to perform the attribute recognition method based on cross-domain assisted training as described in any one of claims 1 to 3.

Images