A method, apparatus, electronic device, and storage medium for target object wearable recognition.

Abstract

This application relates to the field of image processing technology, and more particularly to a method, apparatus, electronic device, and storage medium for identifying whether a target object is wearing a target object, thereby improving the accuracy of identifying whether a target object is wearing a target object. The method includes: performing target detection on an image to be identified to obtain a target region image containing the target object; extracting features from the target object in the target region image to obtain target object features; determining the similarity between auxiliary features and target object features; and determining whether the target object is wearing a target object based on the similarity between the target object features and pre-obtained auxiliary features. Because this application determines whether a target object is wearing a target object based on the similarity between the target object features and pre-obtained auxiliary features, it can adapt to different scenarios for identifying whether a target object is wearing a target object, thus improving the recognition accuracy.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, electronic device and storage medium for target object wearable recognition. Background Technology

[0002] With the popularization of intelligent technologies, the demand for compliance in human body clothing is gradually increasing across various industries. Taking gloves as an example, the health industry requires protective gloves in certain locations, the power industry requires insulated gloves to be worn during operations, and the fire protection industry requires fire gloves to be worn during operations. By standardizing the clothing and attire of relevant personnel, production and operational accidents caused by violations can be effectively reduced.

[0003] In related technologies, deep learning models are mainly used to detect objects in video images to determine whether people in the video images are wearing target objects. However, when faced with new scenarios, the above methods are affected by robustness and generalization, resulting in low accuracy in identifying whether people are wearing target objects. Taking gloves as an example, when the hand area in the video image differs significantly from the hand area in the training set, it is easy to misjudge whether the people in the video image are wearing gloves.

[0004] Therefore, improving the accuracy of identifying whether a target object is wearing the target object is an urgent problem to be solved. Summary of the Invention

[0005] This application provides a target object wear recognition method, device, electronic device, and storage medium to improve the accuracy of recognizing whether a target object is wearing a target object.

[0006] This application provides a target object wearable recognition method, including:

[0007] Perform target detection on the image to be recognized to obtain an image of the target region containing the target object;

[0008] Feature extraction is performed on the target object in the target region image to obtain the target object features;

[0009] Determine the similarity between auxiliary features and the features of the target object; the auxiliary features are obtained at least based on feature extraction of sample objects in a first type of sample images, the first type of sample images including N categories of sub-sample images, the category of the sub-sample images being determined based on whether the sample objects in the sub-sample images are wearing the target object, where N is a positive integer;

[0010] Based on the similarity, it is determined whether the target object is wearing the target object.

[0011] In one optional implementation, N is an integer greater than 1; the auxiliary features include: sub-auxiliary features corresponding to each category of sub-sample images in the N category sub-sample images, the sub-auxiliary features being obtained based on feature extraction of sample objects in the corresponding category sub-sample images;

[0012] Determining the similarity between the auxiliary features and the target object features includes:

[0013] Determine the sub-similarity between the target object features and the sub-auxiliary features corresponding to each category of sub-sample images;

[0014] Determining whether the target object is wearing the target object based on the similarity includes:

[0015] Based on the determined sub-similarity values, it is determined whether the target object is wearing the target object.

[0016] In one optional implementation, the sub-auxiliary feature is obtained based on feature extraction of sample objects in at least two target sample images, wherein the target sample images are sample images in the category sub-sample images corresponding to the sub-auxiliary feature.

[0017] In one optional implementation, the N-category subsample images include at least one of a first characterization image and a second characterization image;

[0018] The sample object in the first characterization image wears the target object;

[0019] The sample object in the second characterization image is not wearing the target object.

[0020] In one optional implementation, the step of extracting features from the target object in the target region to obtain target object features includes:

[0021] The target region image is input into a trained feature extraction model, and the feature extraction model is used to extract features of the target object in the target region image to obtain the features of the target object.

[0022] In one optional implementation, the feature extraction model is trained based on a first training sample dataset, which includes at least the first class of sample images.

[0023] The first training sample dataset also includes a second type of sample images, which are: sample images representing the target object wearing different styles.

[0024] In one alternative implementation, the first training sample dataset is obtained in the following manner:

[0025] The initial sample image and the corresponding segmented image are input into the encoding layer of the trained style transfer model for encoding to obtain the content features of the initial sample image and the segmentation features of the segmented image. The segmented image is a foreground region image containing the sample object obtained by pixel separation of the initial sample image.

[0026] The content features and the segmentation features are input into the feature fusion layer of the style transfer model for feature fusion to obtain the first fused feature;

[0027] The first fusion feature and the target transfer image are input into the style transfer layer of the style transfer model for style transfer to obtain first training sample data with the same wearing style as the target transfer image.

[0028] In one optional implementation, the wearing style includes at least one of wearing status style and wearing style style;

[0029] If the style transfer model is a style transfer model for performing wear state style transfer, then the target transfer image is: a reference image representing the target wear state style of the sample object in relation to the target object, the first training sample data includes the first type of sample images, and the wear state style is used to represent the state of the sample object wearing the target object;

[0030] If the style transfer model is a style transfer model for performing wear style transfer, then the target transfer image is: a reference image of the target object representing the style of the sample object wearing the target style, the first training sample data includes the second type of sample images, and the wear style is used to represent the style of the sample object wearing the target object.

[0031] In one alternative implementation, the trained style transfer model is obtained by:

[0032] The style transfer model to be trained is iteratively trained based on the second training sample dataset, and the trained style transfer model is output. Each second training sample in the second training sample dataset includes the corresponding true class. During one iteration, the following operations are performed:

[0033] The second fusion feature of the second training sample data is flattened to obtain the second fusion feature of the target dimension.

[0034] Based on the feature mapping layer of the style transfer model, feature mapping is performed on the second fusion feature of the target dimension to obtain the predicted category of the second training sample data;

[0035] The style transfer model to be trained is parameter-adjusted using a category loss function constructed based on the difference between the predicted category and the true category of the second training sample data.

[0036] In one optional implementation, if the style transfer model is a style transfer model for performing style transfer of wearing state, then the predicted category and the true category are: a wearing category representing whether the sample object is wearing the target object; if the style transfer model is a style transfer model for performing style transfer of wearing style, then the predicted category and the true category are: a style category representing whether the sample object is wearing the target object.

[0037] In an optional implementation, the sub-auxiliary features corresponding to each category of sub-sample images in the N categories of sub-sample images are obtained in the following manner:

[0038] For each category of subsample images, perform the following operations:

[0039] For a category subsample image, at least two target sample images in the category subsample image are input into a trained feature extraction model to obtain the target sample object features corresponding to each of the at least two target sample images;

[0040] Based on the similarity between the features of each target sample object, the features of each target sample object are divided into a preset number of feature sets;

[0041] The average feature of each target sample object in each feature set is determined, and at least one of the obtained average features is used as the sub-auxiliary feature corresponding to the sub-sample image of the category.

[0042] In one alternative implementation, the feature extraction model is trained in the following manner:

[0043] The feature extraction model to be trained is iteratively trained based on the first training sample dataset, and the trained feature extraction model is output; wherein, in one iteration, the following operations are performed:

[0044] Input at least two selected first training sample data into the feature extraction model to be trained to obtain the first training sample object features corresponding to each of the at least two first training sample data.

[0045] The parameters of the feature extraction model to be trained are adjusted by using a similarity loss function constructed based on the similarity between the features of each first training sample object.

[0046] In one alternative implementation, the similarity loss function is constructed as follows:

[0047] For a single sample object feature, the following operations are performed: the single sample object feature is combined with each of the other sample object features to obtain at least one feature group, and each feature group includes two sample object features;

[0048] The predicted similarity between the features of two sample objects in each feature group is obtained, and a similarity loss function is constructed based on the difference between each predicted similarity and the true similarity. The true similarity of each feature group is determined based on the category labels of the sample data corresponding to the features of the two sample objects.

[0049] This application provides a target object wearable recognition device, comprising:

[0050] The detection unit is used to perform target detection on the image to be recognized and obtain an image of the target region containing the target object.

[0051] The extraction unit is used to extract features from the target object in the target region image to obtain the target object features;

[0052] An acquisition unit is used to determine the similarity between auxiliary features and the features of the target object. The auxiliary features are obtained at least based on feature extraction of sample objects in a first type of sample images. The first type of sample images includes N categories of sub-sample images. The category of the sub-sample images is determined based on whether the sample objects in the sub-sample images are wearing the target object. N is a positive integer.

[0053] A determining unit is configured to determine, based on the similarity, whether the target object is wearing the target object.

[0054] Optionally, N is an integer greater than 1; the auxiliary features include: sub-auxiliary features corresponding to each category of sub-sample images in the N category sub-sample images, the sub-auxiliary features being obtained based on feature extraction of sample objects in the corresponding category sub-sample images;

[0055] Determining the similarity between the auxiliary features and the target object features includes:

[0056] Determine the sub-similarity between the target object features and the sub-auxiliary features corresponding to each category of sub-sample images;

[0057] Determining whether the target object is wearing the target object based on the similarity includes:

[0058] Based on the determined sub-similarity values, it is determined whether the target object is wearing the target object.

[0059] Optionally, the sub-auxiliary feature is obtained based on feature extraction of sample objects in at least two target sample images, wherein the target sample image is a sample image in the category sub-sample image corresponding to the sub-auxiliary feature.

[0060] Optionally, the N category subsample images include at least one of a first characterization image and a second characterization image;

[0061] The sample object in the first characterization image wears the target object;

[0062] The sample object in the second characterization image is not wearing the target object.

[0063] Optionally, the extraction unit is specifically used for:

[0064] The target region image is input into a trained feature extraction model, and the feature extraction model is used to extract features of the target object in the target region image to obtain the features of the target object.

[0065] Optionally, the feature extraction model is trained based on a first training sample dataset, which includes at least the first class of sample images;

[0066] The first training sample dataset also includes a second type of sample images, which are: sample images representing the target object wearing different styles.

[0067] Optionally, the first training sample dataset can be obtained in the following ways:

[0068] The initial sample image and the corresponding segmented image are input into the encoding layer of the trained style transfer model for encoding to obtain the content features of the initial sample image and the segmentation features of the segmented image. The segmented image is a foreground region image containing the sample object obtained by pixel separation of the initial sample image.

[0069] The content features and the segmentation features are input into the feature fusion layer of the style transfer model for feature fusion to obtain the first fused feature;

[0070] The first fusion feature and the target transfer image are input into the style transfer layer of the style transfer model for style transfer to obtain first training sample data with the same wearing style as the target transfer image.

[0071] Optionally, the wearing style includes at least one of wearing status style and wearing style style;

[0072] If the style transfer model is a style transfer model for performing wear state style transfer, then the target transfer image is: a reference image representing the target wear state style of the sample object in relation to the target object, the first training sample data includes the first type of sample images, and the wear state style is used to represent the state of the sample object wearing the target object;

[0073] If the style transfer model is a style transfer model for performing wear style transfer, then the target transfer image is: a reference image of the target object representing the style of the sample object wearing the target style, the first training sample data includes the second type of sample images, and the wear style is used to represent the style of the sample object wearing the target object.

[0074] Optionally, the trained style transfer model can be obtained in the following way:

[0075] The style transfer model to be trained is iteratively trained based on the second training sample dataset, and the trained style transfer model is output. Each second training sample in the second training sample dataset includes the corresponding true class. During one iteration, the following operations are performed:

[0076] The second fusion feature of the second training sample data is flattened to obtain the second fusion feature of the target dimension.

[0077] Based on the feature mapping layer of the style transfer model, feature mapping is performed on the second fusion feature of the target dimension to obtain the predicted category of the second training sample data;

[0078] The style transfer model to be trained is parameter-adjusted using a category loss function constructed based on the difference between the predicted category and the true category of the second training sample data.

[0079] Optionally, if the style transfer model is a style transfer model for performing style transfer of wearing state, then the predicted category and the true category are: the wearing category representing whether the sample object is wearing the target object; if the style transfer model is a style transfer model for performing style transfer of wearing style, then the predicted category and the true category are: the style category representing whether the sample object is wearing the target object.

[0080] Optionally, the sub-auxiliary features corresponding to each category of sub-sample images in the N categories of sub-sample images are obtained in the following way:

[0081] For each category of subsample images, perform the following operations:

[0082] For a category subsample image, at least two target sample images in the category subsample image are input into a trained feature extraction model to obtain the target sample object features corresponding to each of the at least two target sample images;

[0083] Based on the similarity between the features of each target sample object, the features of each target sample object are divided into a preset number of feature sets;

[0084] The average feature of each target sample object in each feature set is determined, and at least one of the obtained average features is used as the sub-auxiliary feature corresponding to the sub-sample image of the category.

[0085] Optionally, the feature extraction model is trained in the following way:

[0086] The feature extraction model to be trained is iteratively trained based on the first training sample dataset, and the trained feature extraction model is output; wherein, in one iteration, the following operations are performed:

[0087] Input at least two selected first training sample data into the feature extraction model to be trained to obtain the first training sample object features corresponding to each of the at least two first training sample data.

[0088] The parameters of the feature extraction model to be trained are adjusted by using a similarity loss function constructed based on the similarity between the features of each first training sample object.

[0089] Optionally, the similarity loss function is constructed in the following way:

[0090] For a single sample object feature, the following operations are performed: the single sample object feature is combined with each of the other sample object features to obtain at least one feature group, and each feature group includes two sample object features;

[0091] The predicted similarity between the features of two sample objects in each feature group is obtained, and a similarity loss function is constructed based on the difference between each predicted similarity and the true similarity. The true similarity of each feature group is determined based on the category labels of the sample data corresponding to the features of the two sample objects.

[0092] An electronic device provided in this application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of any of the target object wearable recognition methods described above.

[0093] This application provides a computer-readable storage medium including a computer program. When the computer program is run on an electronic device, the computer program is used to cause the electronic device to perform the steps of any of the above-described target object wearable recognition methods.

[0094] This application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. When the processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any of the above-described target object wearable recognition methods.

[0095] The beneficial effects of this application are as follows:

[0096] The target object wearing recognition method, apparatus, electronic device, and storage medium provided in this application's embodiments, because this application performs target detection on the image to be recognized to obtain a target region image containing the target object, filters out irrelevant region images in the image to be recognized, reduces interference from irrelevant features, and then extracts features from the target object in the target region image to obtain target object features, and then determines the similarity between auxiliary features and target object features, wherein the auxiliary features are obtained by extracting features from sample images that can characterize whether the sample object is wearing the target object, and determines whether the target object is wearing the target object based on the similarity, can adapt to different scenarios to recognize whether the target object is wearing the target object, and improve the accuracy of recognizing whether the target object is wearing the target object.

[0097] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0098] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0099] Figure 1 This is an optional schematic diagram of an application scenario in an embodiment of this application;

[0100] Figure 2 This is a flowchart illustrating the implementation of a target object wearable recognition method in an embodiment of this application.

[0101] Figure 3This is a schematic diagram of the structure of a style transfer model in an embodiment of this application;

[0102] Figure 4 This is a schematic diagram of the training process of a style transfer model in an embodiment of this application;

[0103] Figure 5 This is a schematic diagram of the training process of a feature extraction model in an embodiment of this application;

[0104] Figure 6 This is a flowchart illustrating a target object wearable recognition method according to an embodiment of this application;

[0105] Figure 7 This is a schematic diagram of the structure of a target object wearable recognition device according to an embodiment of this application;

[0106] Figure 8 This is a schematic diagram of the hardware structure of an electronic device using an embodiment of this application;

[0107] Figure 9 This is a schematic diagram of the hardware structure of another electronic device using an embodiment of this application. Detailed Implementation

[0108] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0109] The following describes some of the concepts involved in the embodiments of this application.

[0110] Style transfer model: used to perform style transfer on the initial sample image based on the target transfer image to obtain the target sample image. In this embodiment, two independent style transfer models can be trained separately: one style transfer model is used to perform style transfer between the sample object wearing the target object and the sample object not wearing the target object, for example, transferring the image of the sample object wearing the target object to the image of the sample object not wearing the target object; the other style transfer model is used to perform style transfer between the sample object wearing different styles of target objects, for example, transferring the image of the sample object wearing a white target object to the image of the sample object wearing a black target object.

[0111] Initial sample images: These are the sample images that need to be style transferred using the style transfer model. The initial sample images can be images of the sample object wearing the target object, images of the sample object not wearing the target object, or images of the sample object wearing different styles of target objects. By performing style transfer on the initial sample images, the first training sample dataset can be effectively expanded.

[0112] Target transfer image: refers to the target image that provides style transfer for the initial sample image. For example, if the initial sample image is an image of the sample object wearing the target object, and the target transfer image is an image of the sample object not wearing the target object, then style transfer transforms the initial sample image into an image of the sample object not wearing the target object.

[0113] The first training sample dataset is used for iterative training of the feature extraction model to be trained, and for extracting features from the first training sample data based on the trained feature extraction model to obtain an auxiliary feature set. The first training sample dataset is obtained by expanding the initial sample images through a style transfer model.

[0114] The second training sample dataset is used to iteratively train the style transfer model to be trained, thereby obtaining a trained style transfer model.

[0115] Auxiliary feature set: includes at least one auxiliary feature. By comparing the similarity between the target object feature and the auxiliary feature, it is possible to determine whether the target object is wearing the target object.

[0116] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0117] like Figure 1 The diagram shown is an application scenario illustration of an embodiment of this application. The application scenario diagram includes two terminal devices 110 and one server 120.

[0118] In this embodiment, the terminal device 110 includes, but is not limited to, mobile phones, tablets, laptops, desktop computers, e-book readers, smart voice interaction devices, smart home appliances, and in-vehicle terminals. The terminal device may have a client application related to target object wearable recognition installed. This client application can be software (such as a browser, video software, etc.), or a webpage, mini-program, etc. The server 120 is a backend server corresponding to the software, webpage, mini-program, etc., or a server specifically used for target object wearable recognition; this application does not impose specific limitations. The server 120 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0119] It should be noted that the target object wearable recognition method in this embodiment can be executed by an electronic device, which can be a server 120 or a terminal device 110. That is, the method can be executed by the server 120 or the terminal device 110 alone, or by both the server 120 and the terminal device 110. For example, when executed by both the server 120 and the terminal device 110, the terminal device 110 sends the image to be recognized to the server 120. The server 120 performs target detection on the image to be recognized, obtains a target region image containing the target object, extracts features from the target object in the target region image, obtains the target object features, then determines the similarity between the auxiliary features and the target object features, and based on the similarity, determines whether the target object is wearing the target object. The server 120 then sends information containing whether the target object is wearing the target object to the terminal device 110, so that the terminal device 110 can determine whether the target object is wearing the target object based on the information.

[0120] In one alternative implementation, the terminal device 110 and the server 120 can communicate via a communication network.

[0121] In one alternative implementation, the communication network is a wired network or a wireless network.

[0122] It should be noted that, Figure 1 The examples shown are merely illustrative; in reality, the number of terminal devices and servers is unlimited and is not specifically limited in the embodiments of this application.

[0123] In this application embodiment, when there are multiple servers, the multiple servers can form a blockchain, and the server is a node on the blockchain; as disclosed in the terminal device 110 method in this application embodiment, the auxiliary features involved can be stored on the blockchain.

[0124] Furthermore, the embodiments of this application can be applied to various scenarios, including not only target object wearable recognition scenarios, but also scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0125] The target object wearable recognition method provided by the exemplary embodiments of this application is described below with reference to the accompanying drawings and the application scenarios described above. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way.

[0126] See Figure 2 The diagram shown is a flowchart of an implementation of a target object wearable recognition method provided in this application. Taking a server as the executing entity, the specific implementation process of this method includes the following steps S21-S24:

[0127] S21: The server performs target detection on the image to be recognized and obtains an image of the target region containing the target object;

[0128] Specifically, when performing target detection on the image to be recognized, the target object can be a human body part, such as the hand, head, and feet. After the target object is detected, the target region can be the smallest rectangle or circle containing the target object, or it can be a region with a preset length and width, etc. This application does not make any specific limitations here.

[0129] In this application embodiment, target detection can be performed on the image to be identified based on the YOLO detection model, or it can be performed based on deep learning network detection models such as SSD (Single Shot MultiBox Detector), Faster R-CNN (Faster Region Recurrent Neural Network), GoogleNet (Google Network), ResNet (Deep Residual Network), and DenseNet (Densely Connected Convolutional Networks). This application does not make any specific limitations here.

[0130] S22: The server extracts features of the target object in the target region image to obtain the features of the target object;

[0131] S23: The server determines the similarity between auxiliary features and target object features;

[0132] The auxiliary features are obtained at least based on feature extraction from the sample objects in the first type of sample images. The first type of sample images includes N categories of sub-sample images, where the category is determined by whether the sample objects in the sub-sample images are wearing the target object, and N is a positive integer. For example, if one category of sub-sample images in the first type of sample images consists of sample images of sample objects wearing the target object, then feature extraction from the first type of sample images can obtain the feature representation of the sample objects wearing the target object, i.e., the auxiliary features. Specifically, the auxiliary features and the target object features can be in vector form, and the cosine similarity between the auxiliary features and the target object features is calculated.

[0133] S24: The server determines whether the target object is wearing the target object based on similarity.

[0134] Based on the similarity between auxiliary features and target object features, it can be determined whether the target object is wearing the target object. For example, a similarity threshold can be set; if the similarity between the auxiliary features and the target object features is higher than the threshold, then the target object is determined to be wearing the target object. Alternatively, the auxiliary features can include auxiliary feature 1 and auxiliary feature 2. The similarity between the target object features and auxiliary feature 1 and auxiliary feature 2 is calculated respectively to obtain similarity 1 and similarity 2. If similarity 1 is greater than similarity 2, then the target object is determined to be wearing the target object. It should be noted that the above-described two methods illustrate the process of determining the wearing information of the target object. In fact, other methods can also be used to determine the wearing information of the target object, and this application does not specifically limit them here.

[0135] In this embodiment, target detection is performed on the image to be identified to obtain a target region image containing the target object. Irrelevant regions in the image to be identified are filtered out to reduce interference from irrelevant features. Then, feature extraction is performed on the target object in the target region image to obtain target object features. Then, the similarity between auxiliary features and target object features is determined. The auxiliary features are obtained by feature extraction on sample images that can characterize whether the sample object is wearing the target object. Based on the similarity, it is determined whether the target object is wearing the target object. This method can adapt to different scenarios to identify whether the target object is wearing the target object and improve the accuracy of identifying whether the target object is wearing the target object.

[0136] In one alternative implementation, step S22 is performed in the following manner:

[0137] First, the target region image is input into a trained feature extraction model. Then, the feature extraction model extracts features of the target object in the target region image to obtain the target object features.

[0138] In one optional implementation, the feature extraction model is trained on a first training sample dataset. The first training sample dataset includes at least a first class of sample images and may also include a second class of sample images. The second class of sample images represents sample images of target objects worn by sample objects in different styles. These different styles can refer to different colors of the target object, such as red, white, black, and blue, or different materials, such as plastic, wool, and cotton / linen. For example, when the target object is a hand and the target object is a glove, the first class of sample images includes images of the sample object wearing gloves and images of the sample object not wearing gloves, and the second class of sample images includes images of the sample object wearing gloves of different colors. The feature extraction model can be a SimCLRv2 (Visual Representation Contrast Learning Framework) model, or other models that can be used for feature extraction, such as the MoCov2 model; no specific limitation is made here.

[0139] In this embodiment, since the first training sample dataset may include images of the sample object wearing the target object and images of the sample object not wearing the target object, or images of the sample object wearing different styles of target objects, or all of the above images, the training data is rich. Therefore, the feature extraction model trained by the first training sample dataset can extract the features of the target object more accurately.

[0140] The following section describes the target object wearing recognition method in this application, using the hand as the target object and the glove as the target glove.

[0141] In one alternative implementation, the first training sample dataset is obtained through the following steps:

[0142] Step 1: Input the initial sample image and the corresponding segmented image into the encoding layer of the trained style transfer model for encoding to obtain the content features of the initial sample image and the segmentation features of the segmented image;

[0143] Step 2: Input the content features and segmentation features into the feature fusion layer of the style transfer model to perform feature fusion and obtain the first fused feature;

[0144] Step 3: Input the first fusion feature and the target transfer image into the style transfer layer of the style transfer model to perform style transfer, and obtain the first training sample data with the same wearing style as the target transfer image.

[0145] The segmented image is a foreground region image containing the sample object, obtained by separating pixels from the initial sample image. Taking the sample object as a hand as an example, the segmented image represents the position of the hand in the initial sample image, or it can be a mask of the hand region, i.e., the target position for style transfer. The first training sample data obtained by the style transfer model has the same wearing style as the target transfer image. For example, if the initial sample image is an image of hand 1 without gloves, and the target transfer image is an image of hand 2 without gloves, then the output first training sample data is an image of hand 1 with gloves.

[0146] In one alternative implementation, the wearing style includes at least one of the wearing status style and the wearing style style;

[0147] If the style transfer model is a style transfer model used for wearing state style transfer, then the target transfer image is: a reference image representing the style of the sample object in relation to the target object as the target wearing state, and the first training sample data includes the first class of sample images;

[0148] If the style transfer model is a style transfer model used for clothing style transfer, then the target transfer image is: a reference image of the target object representing the style of the sample object wearing the target style. The first training sample data includes the second type of sample images, and the clothing style is used to represent the style of the sample object wearing the target object.

[0149] Among them, the wearing state style is used to characterize the state of the sample object wearing the target object. If the style transfer model is a style transfer model used for wearing style style transfer, the target transfer image can be an image of the sample object wearing gloves or an image of the sample object not wearing gloves. The wearing style style is used to characterize the style of the sample object wearing the target object. If the style transfer model is a style transfer model used for wearing style style transfer, taking color as an example, the target transfer image can be an image of the sample object wearing a black target object or an image of the sample object wearing a blue target object.

[0150] In this embodiment, two independent style transfer models are trained. One model can transfer the style of gloved images to ungloved images (i.e., a model for mutual conversion between gloved and ungloved data), and the other model can transfer the style between gloved images (i.e., a model for mutual conversion between gloved data). Since there are often more ungloved images than gloved images in practical applications, and the ungloved images are more diverse, the above method can solve the problem of hand pose effects when expanding the training sample dataset. Furthermore, by transferring the style between images with different styles of gloves, the types of gloves in the training sample dataset can be greatly enriched.

[0151] In one alternative implementation, the trained style transfer model is obtained by means of the following method, see [reference]. Figure 3 Here is a schematic diagram of the structure of a style transfer model in an embodiment of this application:

[0152] The style transfer model to be trained is iteratively trained using the second training sample dataset, and the trained style transfer model is output. The implementation process in one iteration is as follows: Figure 4 As shown, it includes the following steps:

[0153] S41: The server flattens the second fusion feature of the second training sample data to obtain the second fusion feature of the target dimension;

[0154] S42: The server performs feature mapping on the second fusion feature of the target dimension based on the feature mapping layer of the style transfer model to obtain the predicted category of the second training sample data;

[0155] S43: The server uses a category loss function constructed based on the difference between the predicted category and the true category of the second training sample data to adjust the parameters of the style transfer model to be trained.

[0156] In this second training sample dataset, each second training sample data includes the corresponding real category. Before step S41, the content features and segmentation features of the second training sample data are obtained through the encoding layer of the style transfer model, and the content features and segmentation features are fused through the feature fusion layer to obtain the second fused features.

[0157] Specifically, the style transfer model is based on the makeup transfer model (Symmetric Semantic-Aware Transformer, SSAT). The style transfer model has the ability to transfer glove styles. The difference between the style transfer model and the original SSAT is that the glove data used in this application is divided into two distinct semantic categories: wearing gloves and not wearing gloves. Therefore, after the feature fusion (FF) module in the SSAT network, feature mapping can be performed through a fully connected layer to convert it into semantic features, i.e., wearing gloves and not wearing gloves, for additional supervised training. In terms of form, after feature mapping, a semantic feature value, 1 or 0, is output.

[0158] Specifically, such as Figure 3As shown, the solid line process represents the original SSAT basic structure, and the dashed line process represents the feature mapping operation. The fused features output by the FF module are flattened by a reshape operation with parameters [batch_size, -1], where batch_size represents the batch size. The fused features are flattened into 1-dimensional features and then input into the feature mapping module. Specifically, the input dimension is the same as the dimension of the flattened fused features, and the output dimension is a fully connected network with a dimension of [batch_size, 2]. That is, the dimension of the output semantic features is 2, which is similar to supervised learning in form, and then optimized by the softmax (normalization) loss function. The supervision information of supervised learning is the labeling information of the current image with (1) and without (0) gloves, forming a combination of unsupervised and supervised learning.

[0159] The original SSAT training belongs to unsupervised learning, which basically follows the training idea of ​​GAN network. The additional semantic information supervision added in this application belongs to supervised learning. The reason why this operation can work is that the features of wearing gloves and not wearing gloves in the latent space are separate. Using the information of wearing gloves and not wearing gloves for supervision can guide the fused features to be more cohesive in their own feature space. Thus, the attention mechanism in the Symmetrical Semantic Corresponding Feature Transfer (SSCFT) module can better highlight the features of different semantics, and thus realize the transformation of wearing gloves and not wearing gloves at the corresponding positions.

[0160] In this embodiment of the application, the initial sample images are augmented by a style transfer model, and the resulting first training sample dataset has more sample data of hand poses, which can solve the influence of multiple hand poses. Furthermore, the style transfer between gloves can greatly enrich the types of gloves in the dataset.

[0161] In one optional implementation, if the style transfer model is a style transfer model for performing style transfer of wearing state, then the predicted category and the true category are: the wearing category representing whether the sample object is wearing the target object; if the style transfer model is a style transfer model for performing style transfer of wearing style, then the predicted category and the true category are: the style category representing whether the sample object is wearing the target object.

[0162] If the style transfer model is a model for converting between data with and without gloves, it can be either the predicted category is that the sample object is wearing gloves and the true category is that the sample object is wearing gloves, or the predicted category is that the sample object is wearing gloves and the true category is that the sample object is not wearing gloves, and so on. If the style transfer model is a model for converting between data with and without gloves, it can be either the predicted category is that the sample object is wearing black gloves and the true category is that the sample object is wearing black gloves, or other forms, and so on.

[0163] When training a model to convert between data with and without gloves, the two types of supervision information (i.e., the true categories) for wearing and not wearing gloves can be represented by 0 and 1. However, when training a model to convert between data with and without gloves, taking different styles as different colors as an example, the corresponding supervision information can be changed to the specific category of glove color. For example, if the glove color is red, white, blue, black, or orange, the supervision information can be represented by 0, 1, 2, 3, and 4, respectively. Taking different styles as different hand postures as an example, the corresponding supervision information is the hand posture category. For example, hand postures include clenched fist and open hand, and the supervision information for clenched fist and open hand is set to 0 and 1, respectively.

[0164] In this embodiment, style transfer technique (SSAT method) is used to expand the dataset. On one hand, it can transfer the style of gloved images to ungloved images. For example, initially, if there are images of gloved hands with clenched fists and ungloved hands with open hands, the style transfer model can obtain images of ungloved hands with clenched fists and gloved hands with open hands, thus solving the problem of diverse hand poses. On the other hand, it can transfer styles between different types of gloves. For example, performing style transfer on images of clenched fists with black gloves and open hands with white gloves can obtain images of clenched fists with white gloves and open hands with black gloves, thus solving the problem of diverse glove colors and types. This effectively improves the overall glove recognition performance.

[0165] In one optional implementation, N is an integer greater than 1; the auxiliary features include: sub-auxiliary features corresponding to each category of sub-sample images in the N category sub-sample images, the sub-auxiliary features being obtained based on feature extraction of sample objects in the corresponding category sub-sample images; the similarity between the auxiliary features and the target object features is obtained through the following method to determine whether the target object is wearing the target object:

[0166] First, determine the sub-similarity of the target object features and the corresponding sub-auxiliary features of each category of sub-sample images; then, based on the determined sub-similarity, determine whether the target object is wearing the target object.

[0167] Specifically, taking N=2 as an example, the sub-sample images of the two categories are images of the sample object wearing the target object and images of the sample object not wearing the target object. For example, if the sub-similarity between the target object feature and the sub-auxiliary feature of the sample object wearing the target object is 0.8, and the sub-similarity between the target object feature and the sub-auxiliary feature of the sample object not wearing the target object is 0.1, then it is determined that the target object is wearing the target object.

[0168] In one alternative implementation, the sub-auxiliary feature is obtained based on feature extraction of sample objects in at least two target sample images, where the target sample images are sample images in the category sub-sample images corresponding to the sub-auxiliary feature.

[0169] For example, if the category of the subsample image is sample object wearing target object, then the target sample image is the image of sample object wearing target object. By extracting features from the sample object in the target sample image, the obtained sub-auxiliary features can represent the features of sample object wearing target object.

[0170] In one alternative implementation, the N-category sub-sample images include at least one of a first characterization image and a second characterization image; the sample object in the first characterization image is wearing the target object, and the sample object in the second characterization image is not wearing the target object.

[0171] Specifically, the first representation image can be an image of the sample object wearing gloves, and the second representation image can be an image of the sample object not wearing gloves.

[0172] In one alternative implementation, the sub-auxiliary features corresponding to each category of sub-sample images in the N category sub-sample images are obtained in the following manner:

[0173] For each category of subsample images, perform the following operations:

[0174] For a category subsample image, at least two target sample images in the category subsample image are input into a trained feature extraction model to obtain the target sample object features corresponding to each of the at least two target sample images; based on the similarity between the target sample object features, the target sample object features are divided into a preset number of feature sets; the average feature of each target sample object feature in each feature set is determined, and at least one average feature obtained is used as a sub-auxiliary feature corresponding to a category subsample image.

[0175] Specifically, the preset quantity can be one preset quantity or other preset quantities. For example, if the preset quantity is 1, then the features of each target sample object will be divided into 1 feature set. If the preset quantity is 1 and 2, then the features of the target sample object will first be divided into 1 feature set, and then the features of the target sample object will be divided into 2 feature sets. The feature sets divided by other preset quantities will follow the same principle.

[0176] For example, for a category of subsample images, the target object features can be obtained using two independent K-means clustering algorithms. The centroid number K is set to 1 in the first iteration, resulting in one feature set corresponding to this category of subsample images. The centroid number K is set to 6 in the second iteration, resulting in six feature sets. The average feature of the target object features included in the seven feature sets is then taken, and the seven average features are used as the sub-auxiliary features corresponding to this category of subsample images. Correspondingly, the acquisition method of the sub-auxiliary features corresponding to other categories of subsample objects is the same as in the above embodiment, and will not be repeated here.

[0177] In the embodiments of this application, the sub-auxiliary features are obtained in the above manner. Compared with using the original full feature set, this method can greatly reduce the complex calculations when performing feature matching. Compared with using the average features of the original feature set for matching, this method helps to reduce the randomness brought by the average features, thereby improving the accuracy of overall similarity matching.

[0178] In one optional implementation, the auxiliary features include a first set of auxiliary features and a second set of auxiliary features. The first set of auxiliary features is obtained by feature extraction from at least two first representation images, and the second set of auxiliary features is obtained by feature extraction from at least two second representation images.

[0179] Specifically, the first representation image is the image of the sample object wearing gloves, the second representation image is the image of the sample object not wearing gloves, the first auxiliary feature set can represent the feature of the sample object wearing gloves, and the second auxiliary feature set can represent the feature of the sample object not wearing gloves.

[0180] In one alternative implementation, the auxiliary feature set is obtained in the following manner:

[0181] First, at least two selected first training sample data are input into the trained feature extraction model to obtain the first training sample object features corresponding to each of the at least two first training sample data. Then, based on the similarity between the first training sample object features, each first training sample object feature is divided into a preset number of feature sets. Finally, the average feature of each first training sample object feature in each feature set is determined, and at least one obtained average feature is added to the auxiliary feature set as an auxiliary feature.

[0182] In this embodiment, if the first training sample data is a first representation image, then the auxiliary feature is the first auxiliary feature, and the auxiliary feature set is the first auxiliary feature set. If the first training sample data is a second representation image, then the auxiliary feature is the second auxiliary feature, and the auxiliary feature set is the second auxiliary feature set. The preset quantity can be one or more. In this application, we will use two feature divisions with preset quantities of 1 and 6 as examples. Specifically, when the first training sample data is the first representation image, the preset quantity is set to 1 for the first time to obtain feature set 1. The average feature 1 of each first training feature in feature set 1 is added to the auxiliary feature set as an auxiliary feature. The preset quantity is set to 6 for the second time to obtain feature sets 2-7. The average feature 2 of feature set 2, ..., the average feature 7 of feature set 7 are all added to the auxiliary feature set as auxiliary features. At this time, the auxiliary feature set includes features 1-7, which are features representing the target object worn by the sample object. Correspondingly, the process of obtaining the second auxiliary feature set is the same as described in the above embodiment and will not be repeated here.

[0183] For example, all data on wearing gloves is input into a feature extraction model to obtain a set of N-dimensional feature vectors for all gloved data. Based on this set, two independent K-means clustering algorithms are used, with the centroid number K set to 1 in the first instance and 6 in the second instance, to obtain an auxiliary feature set (G1, G6) for gloved data. The basic process for not wearing gloves is similar, resulting in an auxiliary feature set (g1, g6) for not wearing gloves.

[0184] In the embodiments of this application, the auxiliary feature set is obtained in the above manner. Compared with using the original full feature set, this method can greatly reduce the complex calculations when performing feature matching. Compared with using the average features of the original feature set for matching, this method helps to reduce the randomness brought by the average features, thereby improving the accuracy of overall similarity matching.

[0185] In one alternative implementation, step S23 is performed in the following manner:

[0186] The first similarity between each first auxiliary feature and the target object feature is determined, and the mean of each first similarity is taken as the first average similarity; and the second similarity between each second auxiliary feature and the target object feature is determined, and the mean of each second similarity is taken as the second average similarity.

[0187] Step S24 shall be implemented in the following manner:

[0188] If the first average similarity is not less than the second average similarity, then it is determined that the target object is wearing the target object; if the first average similarity is less than the second average similarity, then it is determined that the target object is not wearing the target object.

[0189] Specifically, if the first auxiliary feature set includes first auxiliary features 1-7, then the similarity between the first auxiliary features 1-7 and the target object features is calculated respectively to obtain first similarity 1-7, and the average of the first similarity 1-7 is used as the first average similarity. If the second auxiliary feature set includes first auxiliary features 1-7, then the similarity between the second auxiliary features 1-7 and the target object features is calculated respectively to obtain second similarity 1-7, and the average of the second similarity 1-7 is used as the second average similarity. If the first average similarity is not less than the second average similarity, then it is determined that the target object is wearing the target object; if the first average similarity is less than the second average similarity, then it is determined that the target object is not wearing the target object.

[0190] For example, the average similarity between the target object features and the gloved cluster features (G1, G6) is calculated and denoted as α. The average similarity between the target object features and the ungloved cluster features (g1, g6) is calculated and denoted as β. α - β yields the relative average similarity. When the relative average similarity is greater than or equal to 0, the current hand region features are considered more similar to the gloved features, and therefore the result is determined to be gloved, with an output of (1). Otherwise, the result is determined to be ungloved, with an output of (0).

[0191] In this embodiment, by inputting images of objects wearing gloves into a feature extraction model to obtain auxiliary features for gloves being worn, and images of objects not wearing gloves into a feature extraction model to obtain auxiliary features for not wearing gloves, the system determines whether the target object is wearing gloves based on which auxiliary feature has a higher similarity to the target object's features. This reduces the randomness introduced by average features and improves recognition accuracy. Using relative average similarity for the gloves-wearing determination output avoids the randomness of manually setting thresholds, thereby improving overall detection performance.

[0192] In one alternative implementation, the feature extraction model is trained in the following manner:

[0193] The feature extraction model to be trained is iteratively trained based on the first training sample dataset, and the trained feature extraction model is output; wherein, in one iteration, the following is executed: Figure 5 The implementation process shown includes the following steps:

[0194] S51: The server inputs at least two selected first training sample data into the feature extraction model to be trained, and obtains the first training sample object features corresponding to each of the at least two first training sample data.

[0195] S52: The server uses a similarity loss function constructed based on the similarity between the features of each first training sample object to adjust the parameters of the feature extraction model to be trained.

[0196] Specifically, unlike the original self-supervised training method, this scheme no longer treats the same image and its data-enhanced paired image as the only positive sample pair in the current batch when processing each batch of samples. Instead, it directly optimizes the samples with the same label in the same batch using the cosine similarity loss function based on the labeling information of wearing and not wearing gloves in the image.

[0197] In the embodiments of this application, the feature extraction model obtained through self-supervised learning training can bring similar features closer together and separate dissimilar features. This enables the network to form one cluster of gloved data features and another cluster of ungloved data features, and saves the features extracted from the training dataset to form auxiliary features.

[0198] In one alternative implementation, the similarity loss function is constructed as follows:

[0199] For a single sample object feature, perform the following operations: combine the single sample object feature with each of the other sample object features to obtain at least one feature group, and each feature group includes two sample object features;

[0200] The predicted similarity between the features of two sample objects in each feature group is obtained, and a similarity loss function is constructed based on the difference between each predicted similarity and the true similarity. The true similarity of each feature group is determined based on the category labels of the sample data corresponding to the features of the two sample objects.

[0201] For example, given an input batch size n, i.e., for samples [B1, B2, B3…Bn], the feature extraction model outputs features [F1, F2…Fn], where Fn corresponds to Bn, and Fn has a dimension of N. T [F1, F2…Fn], n is calculated. The cosine similarity matrix Fc of n:

[0202]

[0203] in, Let represent the inner product, and T represent the matrix transpose. Each element in the matrix is ​​the similarity value calculated by combining the N-dimensional vectors representing the features of two sample objects after feature extraction. During the training phase, each element's value is equivalent to the prediction made by the network (i.e., the feature extraction model). This value is inaccurate before the network is properly trained and converges, requiring optimization using supervised information. Analogous to a classic CNN classification network, the final output layer also provides the class value of the current image. However, this value requires proper network training to accurately predict the image's class. Using labeled information as supervision optimizes the network, making its output value more accurate.

[0204] In the cosine similarity matrix, the supervision information corresponding to each element is assigned a value of 1 (consistent) or 0 (inconsistent) depending on whether the sample labels are consistent. There is also a simple algorithm to generate the corresponding supervision information matrix. Assume that the labeling information corresponding to this batch of samples [B1, B2, B3…Bn] is wearing gloves (1) not wearing gloves (0), that is, the labeling information matrix is ​​[1, 0, 1…1], denoted as L. Then the supervision information corresponding to Fc should have this characteristic, that is, for the two positions in L where the index of each element in Fc is located, if they are both 1 or both 0, the supervision information is 1, otherwise the supervision information is 0. Therefore, when generating the supervision matrix, the element 0 in L can be replaced with -1 first, and then L can be generated. T L is the supervisory information matrix, and then -1 is replaced with 0.

[0205] Each element in the cosine similarity matrix represents the predicted similarity between the features of two sample objects in each feature group. After obtaining the supervision information matrix, the true similarity of each feature group is obtained. Based on the difference between each predicted similarity and the true similarity, a similarity loss function is constructed, and the model parameters are adjusted.

[0206] In this embodiment, considering that most labeling information basically uses the cumulative natural numbers 0, 1, 2..., and that this application happens to have two types of 0 and 1 labeling information, the true similarity of each feature group is obtained through the above method, without the need to traverse and judge whether two labels are consistent, thus improving the computational efficiency of similarity loss. Using self-supervised learning technology for hand region feature extraction can effectively avoid overfitting to the training scene caused by supervised learning, thereby enhancing the robustness of glove recognition.

[0207] See Figure 6 This is a flowchart illustrating a target object wearable recognition method according to an embodiment of this application, including the following steps:

[0208] S61: Input the image to be recognized;

[0209] S62: Detect whether the image to be identified contains a hand. If yes, proceed to step S63; otherwise, proceed to step S61.

[0210] S63: Extract features from the target region image containing the hand to obtain hand features;

[0211] S64: Calculate the first average similarity between hand features and gloved auxiliary features, and the second average similarity between hand features and ungloved auxiliary features;

[0212] S65: Determine whether the first average similarity is less than the second average similarity. If yes, proceed to step S66; otherwise, proceed to step S61.

[0213] S66: The hands in the image to be identified are not wearing gloves;

[0214] S67: Issue an alarm.

[0215] In this application embodiment, a detection scheme for compliance of glove operation is implemented. The scheme enhances the hand dataset through makeup style transfer and extracts hand features through self-supervised learning. It analyzes and alerts non-compliance events, uses makeup style transfer to enhance multi-pose data of gloved data based on data without gloves, and enhances multi-class and multi-color data based on gloved data. It uses self-supervised learning to extract features and calculates the average similarity based on the stored auxiliary feature data. The judgment result of whether gloves are worn is adaptively obtained based on the relative average similarity.

[0216] Based on the same inventive concept, embodiments of this application also provide a target object wearable recognition device. For example... Figure 7 As shown, this is a schematic diagram of the target object wearable recognition device 700, which may include:

[0217] Detection unit 701 is used to perform target detection on the image to be recognized and obtain an image of the target region containing the target object;

[0218] Extraction unit 702 is used to extract features of target objects in target region images to obtain target object features;

[0219] The acquisition unit 703 is used to determine the similarity between the auxiliary features and the target object features. The auxiliary features are obtained at least based on feature extraction of the sample objects in the first type of sample images. The first type of sample images includes N categories of sub-sample images. The category of the sub-sample images is determined based on whether the sample objects in the sub-sample images are wearing the target object. N is a positive integer.

[0220] The determination unit 704 is used to determine whether the target object is wearing the target object based on similarity.

[0221] Optionally, N is an integer greater than 1; the auxiliary features include: sub-auxiliary features corresponding to each category of sub-sample images in the N category sub-sample images, and the sub-auxiliary features are obtained based on feature extraction of sample objects in the corresponding category sub-sample images;

[0222] The acquisition unit 703 is specifically used for:

[0223] Determine the sub-similarity of the target object features and the corresponding sub-auxiliary features of each category of sub-sample images;

[0224] The determining unit 704 is specifically used for:

[0225] Based on the determined sub-similarity, it is determined whether the target object is wearing the target object.

[0226] Optionally, the sub-auxiliary features are obtained by feature extraction of sample objects in at least two target sample images, where the target sample images are sample images in the category sub-sample images corresponding to the sub-auxiliary features.

[0227] Optionally, the N-category subsample images include at least one of a first characterization image and a second characterization image;

[0228] The sample object in the first representation image is wearing the target object;

[0229] The sample object in the second characterization image is not wearing the target object.

[0230] Optionally, the extraction unit 702 is specifically used for:

[0231] The target region image is input into a trained feature extraction model, which extracts features of the target object in the target region image to obtain the target object features.

[0232] Optionally, the feature extraction model is trained based on a first training sample dataset, which includes at least the first class of sample images;

[0233] The first training sample dataset also includes a second type of sample images, which are sample images representing target objects worn by sample objects in different styles.

[0234] Optionally, the first training sample dataset can be obtained in the following ways:

[0235] The initial sample image and the corresponding segmented image are input into the encoding layer of the trained style transfer model for encoding to obtain the content features of the initial sample image and the segmentation features of the segmented image. The segmented image is the foreground region image containing the sample object obtained by pixel separation of the initial sample image.

[0236] The content features and segmentation features are input into the feature fusion layer of the style transfer model for feature fusion to obtain the first fused feature.

[0237] The first fusion feature and the target transfer image are input into the style transfer layer of the style transfer model for style transfer to obtain the first training sample data with the same wearing style as the target transfer image.

[0238] Optionally, the wearing style includes at least one of the wearing status style and the wearing style style;

[0239] If the style transfer model is a style transfer model used for wearing state style transfer, then the target transfer image is: a reference image representing the wear state style of the sample object relative to the target object. The first training sample data includes the first type of sample images, and the wear state style is used to represent the state of the sample object wearing the target object.

[0240] If the style transfer model is a style transfer model used for clothing style transfer, then the target transfer image is: a reference image of the target object representing the style of the sample object wearing the target style. The first training sample data includes the second type of sample images, and the clothing style is used to represent the style of the sample object wearing the target object.

[0241] Optionally, the trained style transfer model can be obtained in the following ways:

[0242] The style transfer model to be trained is iteratively trained using the second training sample dataset, and the trained style transfer model is output. Each second training sample in the second training sample dataset includes the corresponding true class. During one iteration, the following operations are performed:

[0243] The second fusion feature of the second training sample data is flattened to obtain the second fusion feature of the target dimension.

[0244] The feature mapping layer based on the style transfer model performs feature mapping on the second fusion feature of the target dimension to obtain the predicted category of the second training sample data.

[0245] The parameters of the style transfer model to be trained are adjusted by using a category loss function constructed based on the difference between the predicted category and the true category of the second training sample data.

[0246] Optionally, if the style transfer model is a style transfer model used for style transfer of wearing state, then the predicted category and the true category are: the wearing category representing whether the sample object is wearing the target object; if the style transfer model is a style transfer model used for style transfer of wearing style, then the predicted category and the true category are: the style category representing whether the sample object is wearing the target object.

[0247] Optionally, the sub-auxiliary features corresponding to each category of sub-sample images in the N categories of sub-sample images are obtained in the following way:

[0248] For each category of subsample images, perform the following operations:

[0249] For a category subsample image, at least two target sample images in the category subsample image are input into a trained feature extraction model to obtain the target sample object features corresponding to each of the at least two target sample images;

[0250] Based on the similarity between the features of each target sample object, the features of each target sample object are divided into a preset number of feature sets;

[0251] The average feature of each target sample object in each feature set is determined, and at least one of the obtained average features is used as the sub-auxiliary feature corresponding to the sub-sample image of the category.

[0252] Optionally, the feature extraction model is trained in the following way:

[0253] The feature extraction model to be trained is iteratively trained based on the first training sample dataset, and the trained feature extraction model is output. During each iteration, the following operations are performed:

[0254] Input at least two selected first training sample data into the feature extraction model to be trained to obtain the first training sample object features corresponding to each of the at least two first training sample data.

[0255] The parameters of the feature extraction model to be trained are adjusted by using a similarity loss function constructed based on the similarity between the features of each first training sample object.

[0256] Optionally, the similarity loss function is constructed as follows:

[0257] For a single sample object feature, perform the following operations: combine the single sample object feature with each of the other sample object features to obtain at least one feature group, and each feature group includes two sample object features;

[0258] The predicted similarity between the features of two sample objects in each feature group is obtained, and a similarity loss function is constructed based on the difference between each predicted similarity and the true similarity. The true similarity of each feature group is determined based on the category labels of the sample data corresponding to the features of the two sample objects.

[0259] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0260] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0261] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device. In one embodiment, the electronic device may be a server, such as... Figure 1 The server 120 is shown. In this embodiment, the structure of the electronic device can be as follows: Figure 8 As shown, it includes a memory 801, a communication module 803, and one or more processors 802.

[0262] The memory 801 is used to store computer programs executed by the processor 802. The memory 801 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0263] Memory 801 may be volatile memory, such as random-access memory (RAM); memory 801 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 801 may be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 801 may be a combination of the above-described memories.

[0264] The processor 802 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 802 is used to implement the aforementioned target object wearable recognition method when it calls the computer program stored in the memory 801.

[0265] The communication module 803 is used to communicate with terminal devices and other servers.

[0266] This application embodiment does not limit the specific connection medium between the memory 801, communication module 803, and processor 802 described above. This application embodiment... Figure 8 The memory 801 and the processor 802 are connected via a bus 804, and the bus 804 is in Figure 8 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. The 804 bus can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 8 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0267] The memory 801 stores a computer storage medium, which stores computer-executable instructions for implementing the target object wearable recognition method of this application embodiment. The processor 802 is used to execute the above-described target object wearable recognition method, such as... Figure 2 As shown.

[0268] In another embodiment, the electronic device can also be other electronic devices, such as... Figure 1 The terminal device 110 is shown. In this embodiment, the electronic device can be structured as follows: Figure 9 As shown, it includes components such as: communication component 910, memory 920, display unit 930, camera 940, sensor 950, audio circuit 960, Bluetooth module 970, and processor 980.

[0269] The communication component 910 is used to communicate with the server. In some embodiments, it may include a Wireless Fidelity (WiFi) module, which is a short-range wireless transmission technology. Electronic devices can use the WiFi module to help users send and receive information.

[0270] The memory 920 can be used to store software programs and data. The processor 980 executes various functions of the terminal device 110 and performs data processing by running the software programs or data stored in the memory 920. The memory 920 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. The memory 920 stores an operating system that enables the terminal device 110 to run. In this application, the memory 920 may store the operating system and various application programs, and may also store a computer program that executes the target object wearable recognition method of the embodiments of this application.

[0271] The display unit 930 can also be used to display information input by the user or information provided to the user, as well as various menus of the terminal device 110, forming a graphical user interface (GUI). Specifically, the display unit 930 may include a display screen 932 disposed on the front of the terminal device 110. The display screen 932 may be configured as a liquid crystal display, a light-emitting diode, or the like. The display unit 930 can be used to display the target object wearable recognition user interface, etc., as described in the embodiments of this application.

[0272] The display unit 930 can also be used to receive input digital or character information and generate signal inputs related to user settings and function control of the terminal device 110. Specifically, the display unit 930 may include a touch screen 931 disposed on the front of the terminal device 110, which can collect touch operations of the user on or near it, such as clicking a button, dragging a scroll box, etc.

[0273] The touchscreen 931 can be placed on top of the display screen 932, or the touchscreen 931 and the display screen 932 can be integrated to realize the input and output functions of the terminal device 110. After integration, it can be referred to as a touch display screen. In this application, the display unit 930 can display the application program and the corresponding operation steps.

[0274] Camera 940 can be used to capture still images, which users can then post comments on via an application. There can be one or multiple cameras 940. An object is projected onto a photosensitive element through a lens, generating an optical image. This photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to a processor 980 to be converted into a digital image signal.

[0275] The terminal device may also include at least one sensor 950, such as an accelerometer 951, a proximity sensor 952, a fingerprint sensor 953, and a temperature sensor 954. The terminal device may also be equipped with other sensors such as a gyroscope, barometer, hygrometer, thermometer, infrared sensor, light sensor, and motion sensor.

[0276] Audio circuitry 960, speaker 961, and microphone 962 provide an audio interface between the user and terminal device 110. Audio circuitry 960 converts received audio data into electrical signals, which are then transmitted to speaker 961, where they are converted into sound signals for output. Terminal device 110 may also be equipped with volume buttons for adjusting the volume of the sound signal. On the other hand, microphone 962 converts collected sound signals into electrical signals, which are received by audio circuitry 960, converted into audio data, and then output to communication component 910 for transmission to, for example, another terminal device 110, or to memory 920 for further processing.

[0277] The Bluetooth module 970 is used to interact with other Bluetooth devices that also have a Bluetooth module via the Bluetooth protocol. For example, a terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smartwatch) that also has a Bluetooth module through the Bluetooth module 970, thereby exchanging data.

[0278] The processor 980 is the control center of the terminal device, connecting various parts of the terminal through various interfaces and lines. It executes various functions and processes data by running or executing software programs stored in the memory 920 and calling data stored in the memory 920. In some embodiments, the processor 980 may include one or more processing units; the processor 980 may also integrate an application processor and a baseband processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the baseband processor mainly handles wireless communication. It is understood that the baseband processor may not be integrated into the processor 980. In this application, the processor 980 can run an operating system, applications, user interface display and touch response, as well as the target object wearable recognition method of this embodiment. Furthermore, the processor 980 is coupled to the display unit 930.

[0279] In some possible implementations, various aspects of the target object wearable recognition method provided in this application can also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program is used to cause the electronic device to perform the steps in the target object wearable recognition method according to the various exemplary embodiments of this application described above. For example, the electronic device can perform actions such as... Figure 2 The steps are shown in the figure.

[0280] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0281] The program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0282] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0283] Computer programs contained on readable media may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0284] Computer programs for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The computer program can execute entirely on the user's electronic device, partially on the user's device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0285] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0286] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0287] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing a computer-usable computer program.

[0288] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce a mechanism for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0289] These computer program commands may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the commands stored in the computer-readable storage medium produce an article of manufacture including command means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0290] These computer program commands can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing the commands executed on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0291] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0292] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A target object wearing recognition method characterized by, The method comprises: target detection is performed on the to-be-identified image to obtain a target region image containing a target object; feature extraction is performed on the target object in the target region image to obtain a target object feature; a similarity between an auxiliary feature and the target object feature is determined; the auxiliary feature is obtained based at least on feature extraction performed on sample objects in first-type sample images, the first-type sample images include N-category sub-sample images, the category of the sub-sample image is determined based on whether the sample object in the sub-sample image wears the target object, and N is an integer greater than 1; the auxiliary feature includes a sub-auxiliary feature corresponding to each of the N-category sub-sample images, and the sub-auxiliary feature is obtained based on feature extraction performed on sample objects in at least two target sample images; the target sample image is a sample image in the sub-auxiliary feature corresponding category sub-sample image; the determination of the similarity between the auxiliary feature and the target object feature comprises: determining a sub-similarity between the target object feature and the sub-auxiliary feature corresponding to each of the category sub-sample images; based on the similarity, determining whether the target object wears the target object; the determination of whether the target object wears the target object based on the similarity comprises: based on the determined sub-similarity, determining whether the target object wears the target object; the feature extraction performed on the target object in the target region to obtain the target object feature comprises: inputting the target region image into a trained feature extraction model, performing feature extraction on the target object in the target region image through the feature extraction model to obtain the target object feature; the feature extraction model is trained based on a first training sample data set, and the first training sample data set at least includes at least one of a first-type sample image and a second-type sample image, and the second-type sample image is a sample image representing a sample object wearing a target object of different styles; the feature extraction model is trained in the following manner: performing cyclic iteration training on a to-be-trained feature extraction model according to the first training sample data set to output the trained feature extraction model; wherein, in one cycle iteration process, the following operations are performed: inputting at least two selected first training sample data into the to-be-trained feature extraction model to obtain first training sample object features corresponding to the at least two first training sample data respectively; performing parameter adjustment on the to-be-trained feature extraction model by using a similarity loss function constructed based on the similarity between the first training sample object features; the similarity loss function is constructed in the following manner: for one sample object feature, the following operations are performed respectively: combining the one sample object feature with each sample object feature to obtain at least one feature group, and each feature group includes two sample object features; The predicted similarity between two sample object features in each feature group is obtained respectively, and a similarity loss function is constructed based on the difference between each predicted similarity and a real similarity, wherein the real similarity of each feature group is determined based on whether the class labels of the sample data corresponding to the two sample object features are consistent.

2. The method of claim 1, wherein, The N-class sub-sample image includes at least one of a first representation image and a second representation image; The sample object in the first representation image wears the target object; The sample object in the second representation image does not wear the target object.

3. The method of claim 1, wherein, The feature extraction model is trained based on a first training sample data set, and the first training sample data set at least includes the first-class sample image; The first training sample data set further includes a second-class sample image, and the second-class sample image is a sample image representing the sample object wearing the target object in different styles.

4. The method of claim 3, wherein, The first training sample data set is obtained in the following manner: The initial sample image and the corresponding segmentation image are input into the encoding layer of the trained style transfer model to obtain the content feature of the initial sample image and the segmentation feature of the segmentation image, and the segmentation image is a foreground region image containing a sample object obtained by pixel separation on the initial sample image; The content feature and the segmentation feature are input into the feature fusion layer of the style transfer model to obtain a first fusion feature; The first fusion feature and a target transfer image are input into the style transfer layer of the style transfer model to perform style transfer and obtain a first training sample data having the same wearing style as the target transfer image.

5. The method of claim 4, wherein, The wearing style includes at least one of a wearing state style and a wearing style style; If the style transfer model is a style transfer model for wearing state style transfer, the target transfer image is a reference image representing the sample object in a target wearing state style for the target object, the first training sample data includes the first-class sample image, and the wearing state style is used to represent the state of the sample object wearing the target object. If the style transfer model is a style transfer model for wearing style style transfer, the target transfer image is a reference image representing the sample object wearing the target object in a target style style, the first training sample data includes the second-class sample image, and the wearing style style is used to represent the style of the sample object wearing the target object.

6. The method of claim 4, wherein, The trained style transfer model is obtained in the following manner: According to the second training sample data set, the style transfer model to be trained is iteratively trained, and the trained style transfer model is output, each second training sample data in the second training sample data set includes a corresponding real class, and in one cycle iteration process, the following operations are performed: The second fusion feature of the second training sample data is flattened to obtain a second fusion feature of a target dimension; mapping the second fusion feature of the target dimension based on a feature mapping layer of the style transfer model to obtain a predicted category of the second training sample data; adopting a category loss function constructed based on a difference between the predicted category and a real category of the second training sample data to adjust parameters of the style transfer model to be trained.

7. The method of claim 6, wherein, If the style transfer model is a style transfer model used for performing style transfer of a wearing state, the predicted category and the real category are a wearing category representing whether the sample object wears the target object. If the style transfer model is a style transfer model used for performing style transfer of a wearing style, the predicted category and the real category are a style category representing a style in which the sample object wears the target object.

8. The method of claim 1, wherein, The sub-assistant feature corresponding to each of the N category sub-sample images is obtained in the following manner: For each category sub-sample image, the following operations are performed respectively: For a category sub-sample image, input at least two target sample images in the category sub-sample image into a trained feature extraction model to obtain target sample object features corresponding to the at least two target sample images respectively; based on the similarity between the target sample object features, divide the target sample object features into a preset number of feature sets; determine the average feature of each target sample object feature in each feature set respectively, and take the obtained at least one average feature as the sub-assistant feature corresponding to the category sub-sample image.

9. A target object wearing recognition apparatus characterized by comprising: comprising: a detection unit configured to perform target detection on a to-be-recognized image to obtain a target region image containing a target object; an extraction unit configured to perform feature extraction on the target object in the target region image to obtain a target object feature; an acquisition unit configured to determine a similarity between an assistant feature and the target object feature, the assistant feature being obtained based at least on feature extraction on a sample object in a first type of sample image, the first type of sample image including N category sub-sample images, the category of the sub-sample image being determined based on whether a sample object in the sub-sample image wears a target object, the N being an integer greater than 1; the assistant feature including a sub-assistant feature corresponding to each of the N category sub-sample images, the sub-assistant feature being obtained based on feature extraction on a sample object in a corresponding category sub-sample image; the determination of the similarity between the assistant feature and the target object feature includes: determining a sub-similarity between the target object feature and the sub-assistant feature corresponding to each of the category sub-sample images; a determination unit configured to determine whether the target object wears the target object based on the similarity; the determination of whether the target object wears the target object based on the similarity includes: determining whether the target object wears the target object based on the determined sub-similarity. The extraction unit is configured to input the target region image into a trained feature extraction model, perform feature extraction on a target object in the target region image by the feature extraction model, and obtain the target object feature. The feature extraction model is trained based on a first training sample data set. The first training sample data set includes at least one of a first type of sample image and a second type of sample image. The second type of sample image is a sample image representing a sample object wearing a target object of different styles. The feature extraction model is trained in the following manner: The feature extraction model is trained in the following manner: According to the first training sample data set, the feature extraction model to be trained is trained by cyclic iteration, and the trained feature extraction model is output. During one cycle iteration, the following operations are performed: At least two selected first training sample data are input into the feature extraction model to be trained, and the first training sample object features corresponding to the at least two first training sample data are obtained. The similarity loss function based on the similarity between the first training sample object features is used to adjust the parameters of the feature extraction model to be trained. The similarity loss function is constructed in the following manner: For one sample object feature, the following operations are performed: combining the one sample object feature with each sample object feature to obtain at least one feature group, and each feature group includes two sample object features.

10. An electronic device, comprising: The predicted similarity between the two sample object features in each feature group is obtained, and the similarity loss function is constructed based on the difference between the predicted similarity and the true similarity. The true similarity of each feature group is determined based on whether the class labels of the two sample object features corresponding to the sample data are consistent.

11. A computer readable storage medium, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method of any one of claims 1-8.

12. A computer program product, characterised in that, It includes a computer program, and when the computer program runs on an electronic device, the computer program is used to make the electronic device execute the steps of the method of any one of claims 1-8. It includes a computer program, and when the computer program runs on an electronic device, the computer program is used to make the electronic device execute the steps of the method of any one of claims 1-8. It includes a computer program, and when the computer program runs on an electronic device, the computer program is used to make the electronic device execute the steps of the method of any one of claims 1-8.

Images