Video costume detection method, device and equipment

Abstract

The embodiments of the present disclosure disclose a video costume detection method, device and equipment, wherein the method comprises: partitioning a plurality of video frames of a to-be-processed video to determine partition information corresponding to each video frame; detecting each video frame to obtain a detection result corresponding to a target person in each video frame; the detection result comprises person information of the target person in the video frame and costume information corresponding to the target person; based on the partition information corresponding to each video frame, the detection results of the video frames in the same partition are corrected to obtain a target detection result corresponding to each video frame; wherein the target detection results corresponding to the video frames in the same partition are the same for the detection result of the target person.

Description

Technical Field

[0001] This disclosure relates to, but is not limited to, the field of image processing technology, and in particular to a video clothing detection method, apparatus, and device. Background Technology

[0002] With the rapid popularization of the internet and the rise and development of e-commerce, image analysis technology based on computer vision has seen unprecedented development. For clothing images taken by models and ordinary users, the goal is to obtain descriptive information about the clothing, such as category, color, texture, and neckline. This is typically achieved by manually adding various attribute tags to the clothing images to search for related clothing. However, because different users have different perceptions of clothing attributes, it is difficult to determine the standard attributes of the clothing in the images, making it challenging to find relevant clothing. Summary of the Invention

[0003] In view of the above, the present disclosure provides at least one video clothing detection method, apparatus, device, storage medium, and program product.

[0004] The technical solution of this disclosure embodiment is implemented as follows:

[0005] On one hand, embodiments of this disclosure provide a video clothing detection method, the method comprising:

[0006] The video to be processed is divided into multiple video frames, and the partition information corresponding to each video frame is determined.

[0007] Each video frame is detected to obtain a detection result for the target person in each video frame; the detection result includes the person information of the target person in the video frame and the clothing information of the target person.

[0008] Based on the partition information corresponding to each video frame, the detection results of video frames within the same partition are corrected to obtain the target detection result corresponding to each video frame; wherein, the target detection results corresponding to video frames within the same partition are the same for the target person.

[0009] On the other hand, embodiments of this disclosure provide a video clothing detection device, the device comprising:

[0010] The partitioning module is used to partition multiple video frames of the video to be processed and determine the partitioning information corresponding to each video frame.

[0011] The detection module is used to detect each of the video frames and obtain the detection result corresponding to the target person in each video frame; the detection result includes the person information of the target person in the video frame and the clothing information corresponding to the target person;

[0012] The correction module is used to correct the detection results of video frames within the same partition based on the partition information corresponding to each video frame, so as to obtain the target detection result corresponding to each video frame; wherein, the target detection results corresponding to video frames within the same partition are the same for the detection results of the target person.

[0013] In another aspect, embodiments of this disclosure provide a computer device including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the program to implement some or all of the steps in the above-described method.

[0014] In another aspect, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above-described method.

[0015] In another aspect, embodiments of this disclosure provide a computer program including computer-readable code, which, when executed in a computer device, causes a processor in the computer device to perform some or all of the steps in the above-described method.

[0016] In another aspect, embodiments of this disclosure provide a computer program product, the computer program product including a non-transitory computer-readable storage medium storing a computer program, wherein when the computer program is read and executed by a computer, it implements some or all of the steps in the above method.

[0017] In this embodiment, by partitioning the video to be processed into multiple video frames, and after obtaining the detection result of each video frame within a partition, a voting statistics can be performed based on the detection results of each video frame within the partition to obtain the target detection result of each video frame. This stabilizes the detection results within a partition, avoiding inconsistencies in detection results for the same target person or clothing in a video due to changes in lighting, angle, etc., thus improving overall detection accuracy. Compared to person and clothing detection in single-frame videos or images, this embodiment can receive user-uploaded videos and perform detection on multiple video frames, enabling a wider recommendation range.

[0018] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0020] Figure 1 This is a schematic diagram illustrating the implementation process of a video clothing detection method provided in an embodiment of this disclosure;

[0021] Figure 2 This is a schematic diagram illustrating the implementation process of a video clothing detection method provided in an embodiment of this disclosure;

[0022] Figure 3 This is a schematic diagram illustrating the implementation process of a video clothing detection method provided in an embodiment of this disclosure;

[0023] Figure 4 This is a schematic diagram illustrating the implementation process of a video clothing detection method provided in an embodiment of this disclosure;

[0024] Figure 5 This is a schematic diagram illustrating the implementation process of a video clothing detection method provided in an embodiment of this disclosure;

[0025] Figure 6 This is a schematic diagram illustrating the implementation process of a video clothing detection method provided in an embodiment of this disclosure;

[0026] Figure 7 This is an optional interface diagram of the clothing display interface provided in this embodiment of the disclosure;

[0027] Figure 8 This is a schematic diagram of the business process of the apparel shopping guide system provided in this embodiment of the disclosure;

[0028] Figure 9A This is a flowchart illustrating the first retrieval strategy provided in an embodiment of this disclosure;

[0029] Figure 9B This is a flowchart illustrating the second retrieval strategy provided in an embodiment of this disclosure;

[0030] Figure 10 This is a schematic diagram of a clothing shopping guide interface provided in an embodiment of this disclosure;

[0031] Figure 11 This is a schematic diagram of the composition structure of a video clothing detection device provided in an embodiment of the present disclosure;

[0032] Figure 12 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this disclosure. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this disclosure. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0034] In the following description, references to "some embodiments" describe a subset of all possible embodiments; however, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict. The terms "first / second / third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this disclosure.

[0036] This disclosure provides a video clothing detection method, which can be executed by a processor of a computer device. The computer device refers to a device with data processing capabilities, such as a server, laptop, tablet, desktop computer, smart TV, set-top box, or mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device).

[0037] Figure 1 This is a schematic diagram illustrating the implementation process of a video clothing detection method provided in an embodiment of this disclosure, as shown below. Figure 1 As shown, the method includes the following steps S101 to S103:

[0038] Step S101: Divide the multiple video frames of the video to be processed into partitions and determine the partition information corresponding to each video frame.

[0039] In some embodiments, the plurality of video frames can be all the video frames in the video to be processed, or it can be a plurality of video frames extracted from the video frames in the video to be processed based on a preset frame extraction rule. The preset frame extraction rule can be to extract frames according to a preset frame extraction frequency, or to extract frames based on the image quality of the video frames. After frame extraction, a subset of the video frames in the video to be processed can represent the information about people and clothing in the entire video, thereby reducing the computational load to some extent.

[0040] In some embodiments, the method of partitioning multiple video frames of the video to be processed may include: partitioning the multiple video frames based on multiple preset partitioning points corresponding to the video to be processed; wherein, the multiple preset partitioning points may be multiple points evenly distributed based on duration (frame number). In other embodiments, the method of partitioning multiple video frames of the video to be processed may further include: performing scene recognition on the multiple video frames to determine the scene information of each video frame, and grouping a video frame or multiple consecutive video frames with the same scene information as a partition.

[0041] The partition information for each video frame is used to determine which partition the video frame belongs to. A partition can contain one or more video frames. When there are multiple video frames within a partition, these multiple video frames are consecutive.

[0042] Step S102: Detect each video frame to obtain the detection result corresponding to the target person in each video frame; the detection result includes the person information of the target person in the video frame and the clothing information corresponding to the target person.

[0043] In some embodiments, for each of multiple video frames, it is necessary to perform person detection on that video frame separately to obtain the person information of the target person in that video frame. This can be achieved by extracting the person's features from the video frame. If these features match a preset person feature from a preset person feature library, the person information corresponding to the matched preset person feature is used as the person information for that video frame. This person information also includes the target person's position information within the current video frame, i.e., the target person's detection bounding box information.

[0044] In some embodiments, for each of multiple video frames, clothing detection needs to be performed on that video frame to obtain the clothing information of the target person in that video frame. This can be achieved by detecting the bounding box information corresponding to the clothing in the video frame, and then extracting the corresponding clothing image from the video frame based on this bounding box information. Furthermore, based on the clothing image, the attribute information of the clothing is detected to obtain the corresponding clothing attributes, which can be clothing category information. This clothing information includes the clothing detection bounding box and the corresponding clothing attributes.

[0045] Step S103: Based on the partition information corresponding to each video frame, the detection results of video frames within the same partition are corrected to obtain the target detection result corresponding to each video frame; wherein, the target detection results corresponding to video frames within the same partition are the same for the detection results of the target person.

[0046] In some embodiments, considering that the detection results of each video frame within the same partition should be uniform, for each partition, the detection results of each video frame within the partition are corrected based on the partition information of each video frame in that partition to obtain the target detection result of each video frame.

[0047] For example, in a partition consisting of three video frames—a first video frame, a second video frame, and a third video frame—if the target person's information and clothing information corresponding to the first, second, and third video frames are all the same, then the detection results of each video frame are not changed. However, if the detection results of the first video frame include target person A1 and clothing B1, the detection results of the second video frame include target person A1 and clothing B2, and the detection results of the third video frame include target person A2 and clothing B1, then the detection results of the video frames within this partition are different, and the detection results of the video frames within this partition need to be corrected.

[0048] In some embodiments, the detection result with the most videos corresponding to each detection result within a partition can be determined as the target detection result for each video frame within that partition by counting the number of video frames corresponding to each detection result. Specifically, the person information with the most videos corresponding to each person information within a partition is determined as the target person information for each video frame within that partition; similarly, the clothing information (clothing category) with the most videos corresponding to each clothing information (clothing category) within a partition is determined as the target clothing information for each video frame within that partition.

[0049] In other embodiments, the video frame quality of each video frame within the partition can be determined separately, and voting weights can be assigned to the corresponding video frames based on the video frame quality, where higher video frame quality results in higher voting weights. Then, based on the detection results of each video frame and the voting weight of each video frame, the target detection result is obtained.

[0050] In this embodiment, by partitioning the video to be processed into multiple video frames, and after obtaining the detection result of each video frame within a partition, a voting statistics can be performed based on the detection results of each video frame within the partition to obtain the target detection result of each video frame. This stabilizes the detection results within a partition, avoiding inconsistencies in detection results for the same target person or clothing in a video due to changes in lighting, angle, etc., thus improving overall detection accuracy. Compared to person and clothing detection in single-frame videos or images, this embodiment can receive user-uploaded videos and perform detection on multiple video frames, enabling a wider recommendation range.

[0051] Figure 2This is an optional flowchart illustrating a video clothing detection method provided in this disclosure embodiment, which can be executed by a computer device's processor. Based on Figure 1 , Figure 1 S101 in the middle can be updated to S201 to S203, which will combine Figure 2 The steps shown are explained.

[0052] Step S201: Obtain multiple video frames corresponding to the video to be processed.

[0053] In some embodiments, the video to be processed may include multiple original video frames that have a temporal relationship. For example, with a frame rate of 30, a video to be processed with a length of 1 second may include 30 original video frames. The multiple video frames may be multiple original video frames that have a temporal relationship among the video frames to be processed.

[0054] In other embodiments, in order to reduce the amount of data computation, the video to be processed can be processed by extracting frames at a preset frequency to obtain multiple video frames; that is, multiple video frames can be obtained by extracting frames from multiple original video frames at a preset frequency.

[0055] Step S202: Based on the set of adjacent video frames corresponding to each video frame, determine the change category corresponding to each video frame; the change category is used to characterize the degree of change of the video frame relative to its corresponding adjacent video frames.

[0056] In some embodiments, steps S2021 to S2022 can be used to determine the change category corresponding to each video frame based on the set of adjacent video frames corresponding to each video frame.

[0057] Step S2021: For each video frame, at least one video frame adjacent to the video frame is taken as the set of adjacent video frames corresponding to the video frame.

[0058] In some embodiments, for each video frame, during the process of determining the change category of the video frame, at least one video frame adjacent to the video frame can be obtained as the set of adjacent video frames corresponding to the video frame. The at least one video frame adjacent to the video frame can be evenly distributed before the video frame, evenly distributed after the video frame, or distributed before and after the video frame respectively.

[0059] Step S2022: Input the set of adjacent video frames corresponding to the video frame into the trained video segmentation network to obtain the change category corresponding to the video frame.

[0060] In some embodiments, a set of neighboring video frames, including the video frame, can be input into a video segmentation network, which can predict the change category of the video frame based on the changes between video frames in the input set of neighboring video frames.

[0061] The change category may include a first category representing a first degree of change and a second category representing a second degree of change, wherein the second degree of change is greater than the first degree of change. For example, the first category may be a category representing that the video frame has not undergone significant changes in the set of adjacent video frames; the second category may be a category representing that the video frame has undergone significant changes in the set of adjacent video frames.

[0062] In some embodiments, the training process of the video segmentation model may include: acquiring multiple sample video frames and a standard category label corresponding to each sample video frame, wherein the standard category label may include the first category and the second category mentioned above; sequentially acquiring a set of sample video frames from the multiple sample video frames according to a preset sliding window, and inputting the set of sample video frames into an initial video segmentation model to obtain a preset category label corresponding to the set of sample video frames, wherein the preset category label is used to characterize the change category of the intermediate video frames in the set of sample video frames; determining a loss value based on the obtained preset category label and standard category label, and adjusting the model parameters of the initial video segmentation model based on the loss value until the model converges, and determining the trained initial video segmentation model as the trained video segmentation network.

[0063] Step S203: Based on the change category corresponding to each video frame, divide the multiple video frames to obtain the partition information corresponding to each video frame.

[0064] In some embodiments, the change category includes a first category representing a first degree of change and a second category representing a second degree of change. Accordingly, steps S2031 to S2032 can be used to divide the plurality of video frames based on the change category corresponding to each video frame to obtain partition information corresponding to each video frame.

[0065] Step S2031: Based on the temporal relationship corresponding to the multiple video frames, cluster the video frames corresponding to each change category to obtain at least one set of video frames; the set of video frames includes one video frame or at least two consecutive video frames, and the at least two consecutive video frames have the same change category.

[0066] In some embodiments, the resulting set of video frames after clustering may include one video frame, or it may include at least two video frames. Where the set of video frames includes at least two video frames, the at least two video frames are consecutive and have the same variation category.

[0067] For example, if there are 20 video frames, where video frames 1 to 8 belong to the first category, video frames 9 to 11 belong to the second category, video frames 12 to 14 belong to the first category, video frame 15 belongs to the second category, and video frames 16 to 20 belong to the first category; then the first set of video frames includes video frames 1 to 8, the second set of video frames includes video frames 9 to 11, the third set of video frames includes video frame 15, and the fourth set of video frames includes video frames 16 to 20.

[0068] Step S2032: Using the video frame located at the center of the target video frame set as the dividing benchmark, divide the multiple video frames to obtain the partition information corresponding to each video frame; the target video frame set is the video frame set of the second category.

[0069] Based on the above example, if there are 20 video frames, where the 1st to 8th video frames belong to the first category, the 9th to 11th video frames belong to the second category, the 12th to 14th video frames belong to the first category, the 15th video frame belongs to the second category, and the 16th to 20th video frames belong to the first category; then the 10th video frame in the 9th to 11th video frames of the second category can be used as the first dividing criterion, and the 15th video frame in the second category can be used as the second dividing criterion; furthermore, these 20 video frames can be divided into the first partition of the 1st to 9th video frames, the second partition of the 11th to 14th video frames, and the third partition of the 16th to 20th video frames.

[0070] Based on the above embodiments, since the video to be processed is divided into multiple video frames, after obtaining the detection results of each video frame in the partition, voting statistics can be performed based on the detection results of each video frame in the partition to obtain the target detection result of each video frame. In this way, the detection results in the partition can tend to be stable, avoiding inconsistencies in the detection results of the same target person and the same clothing in a video due to changes in lighting, angle, etc., thus improving the overall detection accuracy.

[0071] Figure 3 This is an optional flowchart illustrating a video clothing detection method provided in this disclosure embodiment, which can be executed by a computer device's processor. Based on Figure 1 , Figure 1 S102 in the middle can be updated to S301 to S302, which will combine Figure 3 The steps shown are explained.

[0072] Step S301: For each video frame, perform person detection on the video frame to determine the person information corresponding to the target person in the video frame.

[0073] In some embodiments, the above-mentioned person detection of the video frame can be achieved through steps S3011 to S3014 to determine the person information corresponding to the target person in the video frame.

[0074] Step S3011: Perform face detection on the video frame to determine the face image corresponding to the face in the video frame.

[0075] In some embodiments, step S3011 can be implemented as follows: face detection is performed on the video frame to determine the partial image of the face in the video frame and the key point position corresponding to the face; based on the key point position corresponding to the face, the partial image of the face is calibrated to obtain the face image corresponding to the face.

[0076] Step S3012: Extract facial features from the face image to obtain the face features to be matched corresponding to the face.

[0077] In some embodiments, features can be extracted from the face image based on a preset face feature extraction network to obtain the face features to be matched corresponding to the face.

[0078] Step S3013: Obtain a preset facial feature database, which includes at least one preset person and the person information and multiple facial features corresponding to each preset person.

[0079] In some embodiments, the preset person can be a person who needs to be identified. For example, in a scenario where the implementation is a celebrity clothing recommendation, the preset person in the facial feature database can be a pre-set celebrity object. In another scenario where the implementation is a teacher clothing recommendation, the preset person in the facial feature database can be a pre-set teacher object from the current school.

[0080] In some embodiments, for each preset person in the face feature library, in order to improve the accuracy of person detection in video frames, multiple face features can be set for each preset person. In the process of determining whether the person (face) in the video frame is a preset person in the face feature library, the feature distance between the face feature to be matched corresponding to the face in the video frame and each face feature corresponding to the preset person can be calculated respectively, so as to determine whether the face in the video frame is the preset person.

[0081] Step S3014: Based on the feature distance between the face feature to be matched and the multiple face features corresponding to each preset person, determine the target preset person corresponding to the face among the multiple preset people, and determine the person information corresponding to the target preset person as the person information corresponding to the target person.

[0082] In some embodiments, determining the target preset person corresponding to the face among the plurality of preset persons based on the feature distance between the face feature to be matched and the plurality of face features corresponding to each preset person includes: for each preset person, determining the feature distance between the face feature to be matched and each face feature corresponding to the preset person; comparing the feature distance of each face feature with a preset distance threshold to determine the similarity result corresponding to each face feature; the similarity result is used to characterize whether the face corresponds to the preset person corresponding to the face feature; and determining the target preset person corresponding to the face among the plurality of preset persons based on the similarity result corresponding to each face feature.

[0083] In this scenario, with N preset individuals corresponding to 7 facial features, and each facial feature being a 2048-dimensional feature vector, a 7N×2048 feature matrix can be obtained. Simultaneously, the facial feature to be matched is also a 2048-dimensional feature vector. Performing 7N distance calculations on the obtained facial feature to be matched and the 7N×2048 feature matrix yields a 1×7N distance matrix. Here, 1 refers to the detection of 1 face corresponding to the facial feature to be matched in the current video frame. If M faces are detected, an M×7N distance matrix can be obtained. Based on a preset distance threshold, this 1×7N distance matrix is ​​binarized, i.e., elements greater than the distance threshold are set to the first value, and elements less than the distance threshold are set to the second value, resulting in a 1×7N binary matrix to be verified. This binary matrix includes the similarity result for each facial feature.

[0084] In this case, if all elements in the binary matrix are 0, it means that the person in the video frame is not any of the preset persons in the face feature database.

[0085] In some embodiments, when all the first similarity results correspond to a pre-matched person, the pre-matched person is determined as the target person; the first similarity result indicates that the face corresponds to the pre-matched person whose facial features are corresponding to the pre-matched person.

[0086] If at least two elements with a value of 1 are located in the same interval of a matched preset person, then the same matched preset person is taken as the target preset person corresponding to the facial features to be matched in the current video frame.

[0087] In some embodiments, when the first similarity result corresponds to at least two matched preset persons, the target preset person is determined from the at least two matched preset persons based on the feature distance between the face feature to be matched and the sum of the feature distances between the face feature to be matched and the multiple face features corresponding to each of the matched preset persons.

[0088] In this case, if at least two elements with a value of 1 are located within the intervals of at least two matched preset persons, it is necessary to further determine a target matched person as the target preset person corresponding to the face in the current video frame. Since the vector distance between the same person is small and the vector distance between different people is large, for each of the at least two matched preset persons, the feature distance sum between the face feature to be matched and the multiple face features of each matched preset person can be calculated. Thus, at least two feature distance sums can be obtained, and the matched preset person corresponding to the smallest feature distance sum is taken as the target preset person corresponding to the face feature to be matched in the current video frame.

[0089] Step S302: Perform clothing detection on the video frame to determine the clothing information of the target person.

[0090] In some embodiments, the clothing information includes the clothing category, and the above-mentioned clothing detection of the video frame can be achieved through steps S3021 to S3024 to determine the clothing information corresponding to the target person.

[0091] Step S3021: Extract the clothing feature map corresponding to the video frame.

[0092] Step S3022: Based on the clothing feature map, determine the detection box information corresponding to the clothing in the video frame.

[0093] In some embodiments, the detection box information corresponding to the clothing includes the position of the detection box corresponding to the clothing in the video frame. Specifically, based on the clothing feature map corresponding to the video frame, several anchor boxes (anchor boxes of different sizes are placed around the feature point) are placed at each feature point in the clothing feature map. Each anchor box can predict the position of a candidate box. During training, anchor boxes close to the actual clothing box position are used as positive samples, and others are used as negative samples. This ensures that the predicted box positions corresponding to the anchor boxes of the positive samples are more accurate during testing. The top 100 predicted boxes with the highest scores are selected as candidate boxes. Region pooling is performed on the obtained candidate boxes on the feature map. The pooled features are used to classify the candidate boxes and further optimize them (adjusting the position of the candidate boxes by determining the position offset), ultimately obtaining the category and final position of the clothing box.

[0094] Step S3023: Based on the detection box information corresponding to the clothing, determine the clothing features corresponding to the clothing.

[0095] In some embodiments, step S3023 can be implemented as follows: based on the detection box information corresponding to the clothing, extract the clothing image corresponding to the clothing from the video frame; perform feature extraction on the clothing image to obtain the clothing features corresponding to the clothing.

[0096] In some embodiments, the clothing features include multiple clothing sub-features, and the above step S3023 can also be implemented by the following method: based on the detection box information corresponding to the clothing, extract the clothing image corresponding to the clothing from the video frame; perform data augmentation processing on the clothing image to obtain at least one similar clothing image corresponding to the clothing; perform feature extraction on the clothing image and the at least one similar clothing image to obtain multiple clothing sub-features corresponding to the clothing.

[0097] Specifically, for the specific garment, an image of the garment can be extracted from the video frame based on the corresponding detection bounding box information. Since garment images are single images, and clothing often features wrinkles, deformations, and uneven size proportions, feature capture relying solely on the garment image itself is prone to randomness, chance, and background noise. Therefore, intelligent data augmentation is needed to enhance the network's recognition of garment details, such as texture patterns. This data augmentation process includes at least one of the following: translation, scaling, flipping, and various affine transformations. This allows the network to obtain the garment image corresponding to the garment in the video frame and at least one similar garment image obtained through data augmentation.

[0098] Specifically, for the clothing image and the at least one similar clothing image, the same feature extraction method can be used to extract features from each image to obtain multiple clothing sub-features corresponding to the clothing.

[0099] Step S3024: Determine the clothing category of the clothing based on the clothing characteristics corresponding to the clothing.

[0100] In some embodiments, the clothing category may include a garment category, a texture category, and a main color category. To improve the classification accuracy of the clothing category, a first classifier corresponding to the garment category, a second classifier corresponding to the texture category, and a third classifier corresponding to the main color category can be pre-built. The clothing features (multiple clothing sub-features) corresponding to the clothing are then input into the first classifier, the second classifier, and the third classifier, respectively, to obtain the garment category, texture category, and main color category of the clothing.

[0101] Based on the above embodiments, by performing person detection and clothing detection on each video frame in multiple video frames respectively, clothing information and person information corresponding to each video frame can be obtained, thereby improving the detection accuracy.

[0102] Figure 4 This is an optional flowchart illustrating a video clothing detection method provided in this disclosure embodiment. This method can be executed by a computer device's processor. The clothing information includes data information about the clothing; based on... Figure 3 , Figure 3 S302 in the formula may also include S401 to S403, which will combine Figure 4 The steps shown are explained.

[0103] Step S401: Determine the target retrieval strategy based on the data volume of the preset clothing library in the first retrieval strategy and the second retrieval strategy; the data volume corresponding to the first retrieval strategy is higher than the data volume corresponding to the second retrieval strategy; the preset clothing library includes multiple preset clothing items and data information corresponding to each preset clothing item.

[0104] In some embodiments, a data volume threshold can be set. If the data volume of the preset clothing library is greater than or equal to the data volume threshold, the first search strategy is used as the target search strategy; if the data volume of the preset clothing library is less than the data volume threshold, the second search strategy is used as the target search strategy.

[0105] In some embodiments, the data information corresponding to the preset clothing may include clothing images, clothing names, purchase links, etc.

[0106] Step S402: Using the target retrieval strategy, determine at least one preset garment that matches the garment among the plurality of preset garments.

[0107] In some embodiments, when the target retrieval strategy is the first retrieval strategy, the above-mentioned determination of at least one preset garment matching the garment among the plurality of preset garments can be achieved by using the target retrieval strategy through steps S4021 to S4024.

[0108] Step S4021: Based on the clothing image, determine the clothing feature vector corresponding to the clothing image.

[0109] Step S4022: Among the multiple first central features corresponding to the multiple preset clothing items, determine the target first central feature that matches the clothing feature vector; the multiple first central features are determined by clustering multiple preset clothing vectors in the preset clothing library, and each first clustering result obtained by clustering corresponds to a first central feature.

[0110] Step S4023: Based on the target first central feature, the clothing feature vector is quantized and encoded to obtain the clothing code.

[0111] In some embodiments, the above-mentioned quantization encoding of the clothing feature vector based on the target first central feature to obtain the clothing code can be achieved by the following implementation: Based on different feature positions, the clothing feature vector is split into features within each feature to obtain a clothing sub-vector corresponding to each feature position of the clothing feature vector; based on the second central feature of each feature position corresponding to the target first central feature, the clothing sub-vector corresponding to each feature position is quantized to obtain a quantized value of the clothing sub-vector corresponding to each feature position; the second central feature is determined by clustering the preset sub-vectors of multiple feature positions corresponding to the target first central feature, and each second clustering result corresponds to a second central feature; the clothing code is determined based on the quantized value of the clothing sub-vector corresponding to each feature position.

[0112] Step S4024: Determine the preset clothing corresponding to the preset clothing code corresponding to the target encoding result as at least one preset clothing that matches the clothing; the target encoding result is at least one quantization encoding result that matches the clothing code among multiple quantization encoding results corresponding to the target first central feature; the method for quantizing the preset clothing vector corresponding to the target first central feature is the same as the method for quantizing the clothing feature vector.

[0113] In some embodiments, the above encoding method includes:

[0114] Clustering is performed on multiple preset clothing vectors in the preset clothing library to obtain multiple first clustering results; each first clustering result corresponds to a first central feature;

[0115] For each of the first clustering results, based on different feature positions, each of the preset clothing vectors corresponding to the first clustering results is split into features to obtain clothing sub-features corresponding to each feature position of each preset clothing vector.

[0116] For each of the aforementioned feature locations, the clothing sub-features corresponding to the feature location in each of the first clustering results are clustered to obtain multiple second clustering results; each of the second clustering results corresponds to a second central feature; based on the second central feature corresponding to each of the second clustering results, each of the clothing sub-features corresponding to the feature location is quantized and encoded to obtain the quantized value of each of the clothing sub-features corresponding to the feature location;

[0117] For each of the preset clothing vectors, the quantization encoding result of the preset clothing vector is determined based on the quantization value of each clothing sub-feature in the preset clothing vector.

[0118] In some embodiments, when the target retrieval strategy is the second retrieval strategy, the above-mentioned method of using the target retrieval strategy to determine at least one preset garment that matches the garment among the plurality of preset garments can be achieved through steps S4025 to S4027.

[0119] Step S4025: Based on the clothing image, determine the overall feature vector and at least one local feature vector corresponding to the clothing image.

[0120] Step S4026: For each of the plurality of preset garments, determine the preset overall vector and the overall feature vector corresponding to the preset garment to determine a first similarity, and determine a second similarity between each preset local vector and each local feature vector corresponding to the preset garment; based on the first similarity and at least one second similarity, determine the preset similarity between the preset garment and the garment.

[0121] In some embodiments, a corresponding graph inference network can be constructed based on a first similarity and at least one second similarity. In the graph inference network, one node represents the first similarity, and other nodes represent the second similarities respectively. Each edge represents the relationship between two similarities (the first similarity and the second similarity, and the relationship between the second similarities). Finally, the cross-entropy classification loss function is used to allow the network to judge the nodes of global features to determine the preset similarity between the preset clothing and the preset clothing.

[0122] Step S4027: Based on the preset similarity corresponding to each preset garment, determine at least one preset garment that matches the garment among the plurality of preset garments.

[0123] In some embodiments, at least one preset garment with the highest preset similarity is used as at least one preset garment that matches the garment.

[0124] Step S403: Determine the data information corresponding to each preset garment that matches the garment as the data information of the garment.

[0125] Based on the above embodiments, since the target retrieval strategy is determined in the first retrieval strategy and the second retrieval strategy based on the amount of data in the preset clothing library, and then the preset clothing matching the clothing is retrieved from the preset clothing library based on the target retrieval strategy, and the data information corresponding to the preset clothing is determined as the data information of the clothing, different retrieval strategies can be flexibly selected to improve retrieval efficiency.

[0126] Figure 5 This is an optional flowchart illustrating a video clothing detection method provided in this disclosure embodiment, which can be executed by a computer device's processor. Based on Figure 1 The clothing information of the target person includes the detection box information and clothing category corresponding to each garment in the video frame. Figure 1 S103 in the middle can be updated to S501 to S502, which will combine Figure 5 The steps shown are explained.

[0127] Step S501: For each partition, classify each garment in each video frame within the partition to obtain at least one detection category; the detection box information of the garment corresponding to the detection category in the corresponding at least one video frame to be corrected satisfies a preset overlap condition.

[0128] In some embodiments, based on the detection box information corresponding to each garment in each video frame, all detection boxes within a partition can be classified to determine the number of garments present in each video frame within the current partition, wherein each garment corresponds to a detection category. The detection box information of the garment corresponding to this detection category in at least one corresponding video frame to be corrected satisfies a preset overlap condition. To facilitate understanding of this scheme, let's take the example where each video frame includes one type of garment: Detection box information for one garment present in N video frames within the partition is obtained respectively. Based on a preset classification algorithm, if the intersection-union ratio (IUU) between any two pairs of detection boxes in the N video frames is greater than a preset threshold, then the partition corresponds to a detection category, meaning that only one garment from the real scene exists within the partition. This garment from the real scene has corresponding garment information (detection box information and garment category) in every video frame of the partition.

[0129] For example, in a partition that includes three video frames: a first video frame, a second video frame, and a third video frame, if the detection result of the first video frame includes shirt A1 and pants B1, the detection result of the second video frame includes shirt A1 and pants B2, and the detection result of the third video frame includes shirt A2 and pants B1. In the process of classifying the partition to obtain at least one detection category, it is necessary to classify the six detection box information. If the three shirt detection boxes of the first video frame (shirt A1), the second video frame (shirt A1), and the third video frame (shirt A2) satisfy the preset overlap condition, then these three shirt detection boxes are classified into the first detection category, and the video frames to be corrected corresponding to the first detection category are the first to third video frames. Correspondingly, if the two pants detection boxes of the second video frame (pants B2) and the third video frame (pants B1) satisfy the preset overlap condition, then these two pants detection boxes are classified into the second detection category, and the video frames to be corrected corresponding to the second detection category are the second and third video frames. The pants detection box of the first video frame (pants B1) is classified into the third detection category, and the video frame to be corrected corresponding to the third detection category is the first video frame.

[0130] Step S502: For each garment, determine the target garment category based on the garment category corresponding to the garment in each video frame to be corrected.

[0131] Based on the above example, given the first detection category (three clothing detection boxes for clothing A1 in the first video frame, clothing A1 in the second video frame, and clothing A2 in the third video frame), the second detection category (two clothing detection boxes for pants B2 in the second video frame and pants B1 in the third video frame), and the third detection category (the clothing detection box for pants B1 in the first video frame), the third detection category, since it only has one video frame, retains the clothing detection box for pants in the first video frame, with the clothing category being pants B1. For the first detection category, it is necessary to determine the target clothing category corresponding to these three clothing detection boxes based on clothing A1 in the first video frame, clothing A1 in the second video frame, and clothing A2 in the third video frame, i.e., determine whether the clothing corresponding to the first detection category is clothing A1 or clothing A2. For the second detection category, it is necessary to determine the target clothing category corresponding to these two clothing detection boxes based on clothing B2 in the second video frame and clothing B1 in the third video frame, i.e., determine whether the clothing corresponding to the second detection category is clothing B1 or clothing B2.

[0132] In some embodiments, steps S5021 to S5023 can be used to determine the target clothing category corresponding to the clothing based on the clothing category corresponding to the clothing in each of the video frames to be corrected.

[0133] Step S5021: Obtain the video frame quality corresponding to each video frame to be corrected.

[0134] In some embodiments, obtaining the video frame quality corresponding to each video frame to be corrected includes: for each video frame to be corrected, determining the clothing region corresponding to the video frame to be corrected based on the detection box information corresponding to the clothing; and determining the video frame quality corresponding to the video frame to be corrected based on the video frame to be corrected and the clothing region.

[0135] The video frame quality includes at least one of the following: the degree of occlusion by the clothing, the clarity of the clothing area corresponding to the clothing, and the brightness of the clothing area corresponding to the clothing.

[0136] Step S5022: Based on the video frame quality corresponding to each video frame to be corrected, determine the voting weight corresponding to each video frame to be corrected; the voting weight corresponding to the video frame to be corrected is positively correlated with the video frame quality corresponding to the video frame to be corrected.

[0137] Step S5023: Based on the voting weight corresponding to each video frame to be corrected and the clothing category corresponding to the clothing in each video frame to be corrected, determine the target clothing category corresponding to the clothing.

[0138] In some embodiments, the video frame quality of a video frame can be estimated based on the degree of occlusion of the clothing corresponding to each video frame, the clarity of the clothing area corresponding to the clothing, and the brightness of the clothing area corresponding to the clothing, to obtain a video frame quality quantization value; and the voting weight corresponding to each video frame to be corrected can be determined based on the video frame quality quantization value corresponding to each video frame to be corrected.

[0139] For example, taking the first detection category mentioned above, it is necessary to determine the target clothing category corresponding to the three clothing detection boxes based on clothing A1 in the first video frame, clothing A1 in the second video frame, and clothing A2 in the third video frame. The video frame quality quantization values ​​of the first to third video frames are 2, 2, and 6 respectively, so the voting weights corresponding to the first to third video frames are determined to be 0.2, 0.2, and 0.6 respectively. Combining this with the clothing category in each video frame, the voting results are 0.4 for clothing A1 and 0.6 for clothing A2. Therefore, the clothing category of the three clothing detection boxes corresponding to the first to third video frames is set as the target clothing category (clothing A2).

[0140] In other embodiments, the video frame quality includes quality information for at least one quality dimension, and the clothing category includes at least one clothing subcategory.

[0141] Accordingly, the above-mentioned determination of the voting weight corresponding to each video frame to be corrected based on the video frame quality corresponding to each video frame to be corrected includes: for each clothing sub-category, determining the voting sub-weight corresponding to the clothing sub-category based on the degree of correlation between the clothing sub-category and each quality dimension, and the quality information of each quality dimension.

[0142] Accordingly, the above-mentioned determination of the target clothing category corresponding to the clothing based on the voting weight corresponding to each of the video frames to be corrected and the clothing category corresponding to the clothing in each of the video frames to be corrected includes: for each clothing sub-category, determining the target clothing sub-category corresponding to the clothing based on the voting sub-weight corresponding to the clothing sub-category and the clothing sub-category corresponding to the clothing in each of the video frames to be corrected.

[0143] In some embodiments, the clothing category may include clothing subcategories with different dimensions. For each clothing subcategory, the voting sub-weight corresponding to the clothing subcategory may be determined based on the degree of correlation between the clothing subcategory and each of the corresponding quality dimensions, and the quality information of each of the quality dimensions.

[0144] In some embodiments, taking a clothing category that includes a primary color category as an example, when the video frame quality includes the degree of occlusion of the clothing, the clarity of the clothing area corresponding to the clothing, and the brightness of the clothing area corresponding to the clothing, the primary color category is highly correlated with the brightness of the clothing area, and less correlated with the degree of occlusion and the clarity of the clothing area; taking a clothing category that includes a texture category as an example, the texture category is highly correlated with the degree of occlusion and the clarity of the clothing area, and less correlated with the brightness of the clothing area.

[0145] For example, taking the second detection category mentioned above, based on the pants in the second video frame (hue C1, texture D1) and the pants in the third video frame (hue C2, texture D2), the target clothing category corresponding to these two pants detection boxes is determined, that is, whether the hue of the pants belongs to C1 or C2, and whether the texture belongs to D1 or D2. With the luminance quantization value of 3, the occlusion quantization value of 6, and the sharpness of 6 in the second video frame, and the luminance quantization value of 6, the occlusion quantization value of 3, and the sharpness of 3 in the third video frame, in determining the hue of the pants, the correlation degree can be set to 6, 2, and 2 for luminance, occlusion, and sharpness respectively. Then, the voting result for the hue of the pants belonging to C1 is 42, and the voting result for belonging to C2 is 48, meaning the hue of the pants belongs to C2. Similarly, in determining the texture of the pants, the correlation degree can be set to 2, 4, and 4 for luminance, occlusion, and sharpness respectively. Then, the voting result for the texture of the pants belonging to D1 is 54, and the voting result for belonging to D2 is 36, meaning the texture of the pants belongs to D1. It can be seen that, for the same detection category, that is, for the same clothing, the target clothing subcategory can exist in different video frames to be corrected.

[0146] Based on the above embodiments, since the target detection result of each video frame is obtained by voting statistics based on the detection results of each video frame within the partition, the detection results within the partition can tend to be stable, avoiding inconsistencies in the detection results of the same target person and the same clothing in a video due to changes in lighting, angle, etc., thus improving the overall detection accuracy.

[0147] Figure 6 This is an optional flowchart illustrating a video clothing detection method provided in this disclosure, which can be executed by a computer device's processor. Based on the above embodiments, using... Figure 1 For example, Figure 1 The method may also include S601 to S603, combining Figure 6 The steps shown are explained.

[0148] Step S601: Play the video to be processed through the clothing display interface.

[0149] In some embodiments, the clothing display interface may include a video playback area where the video to be processed is played.

[0150] Step S602: During the process of displaying the target video frame in the video to be processed, display the character information of the target person in the target video frame and the clothing information corresponding to the target person.

[0151] In some embodiments, the target video frame is any one of the multiple original video frames included in the video to be processed. Step S602 uses the display of the target video frame as an example to explain the content displayed in other areas of the clothing display interface. That is, as the video to be processed is played, the target video frame in the video playback area changes accordingly, and the content displayed in other areas also changes with the change of the target video frame.

[0152] In some embodiments, during the display of a target video frame from the video to be processed, the target video frame displays the target person's information and their corresponding clothing information. The aforementioned person and clothing information can be displayed as rectangular boxes showing the location / range of the person and clothing, and as text showing the person's identity information and the clothing's category information.

[0153] In some embodiments, the above-mentioned display of the target person's information and the corresponding clothing information in the target video frame can be achieved through steps S6021 to S6022.

[0154] Step S6021: Display the character information of the target person in the target video frame through the character display area in the clothing display interface.

[0155] In some embodiments, the person information of the target person includes a partial facial image of the target person in the target video frame. Accordingly, the person display area may include multiple person sub-regions. If there are N target people in the target video frame, N person sub-regions can be displayed in the person display area. At the same time, each person sub-region displays a partial facial image of the corresponding target person in the current target video frame.

[0156] In some embodiments, the target person's information includes the target person's identity information.

[0157] Step S6022: Display the clothing information corresponding to the target person in the target video frame through the clothing display area in the clothing display interface; the clothing information includes a partial clothing image and clothing category corresponding to each clothing item in the target video frame.

[0158] In some embodiments, the clothing information corresponding to the target person includes a partial clothing image of the target person in the target video frame. Accordingly, the clothing display area may include multiple clothing sub-regions. If there are N clothing items in the target video frame, N clothing sub-regions can be displayed in the clothing display area. At the same time, each clothing sub-region displays a partial clothing image of the corresponding target person in the current target video frame.

[0159] In some embodiments, the clothing information corresponding to the target person includes the clothing category corresponding to each garment. The clothing category may include, but is not limited to, at least one of the following: shirt, T-shirt, shorts, trousers, skirt, long skirt, dress, jumpsuit, etc.

[0160] In some embodiments, to facilitate users in purchasing the clothing they need, the method may further include steps S603 to S604.

[0161] Step S603: Receive a trigger operation for the target clothing image in the local clothing image corresponding to each of the clothing items.

[0162] In some embodiments, the clothing display interface can receive a trigger operation from a user targeting a target clothing image in a partial clothing image corresponding to at least one clothing item displayed in the current clothing display area; in response to the trigger operation, step S604 is executed. The trigger operation may include a long press, a click, or a voice selection operation, etc.

[0163] Step S604: In response to the triggering operation, at least one related garment and a purchase link corresponding to each related garment are displayed in the garment purchase area of ​​the garment display interface.

[0164] In some embodiments, during the display of the video to be processed, the clothing display interface may not show the clothing purchase area if the trigger operation is not received. In response to a trigger operation targeting a target clothing image corresponding to a target video frame, the clothing purchase area may be displayed, and at least one related garment corresponding to the target clothing image and a purchase link for each of the related garments may be shown through the clothing purchase area.

[0165] In some embodiments, the method may further include receiving a user's selection operation for a target related garment among the at least one related garments through the garment purchase area of ​​the garment display interface, and in response to the selection operation, redirecting to the purchase interface corresponding to the purchase link of the target related garment.

[0166] For example, please refer to Figure 7 , Figure 7 This is an optional schematic diagram of the clothing display interface provided in this embodiment of the disclosure. The clothing display interface 710 may include a character display area 711, a clothing display area 712, and a video playback area 714. In response to a trigger operation on a target clothing image in the clothing display area, at least one related clothing item corresponding to the target clothing image and a purchase link corresponding to each of the related clothing items can be displayed in the clothing purchase area 713.

[0167] Based on the above embodiments, a visual interface can be used to display the information of the target person and the clothing of the target person in the currently displayed target video frame in real time during the process of displaying the video to be processed, thereby improving the correlation between the search results and the original video. At the same time, after receiving a user's click operation on a clothing image, at least one related clothing item corresponding to the clothing image and a purchase link for each of the related clothing items can be further displayed, which can improve the user's interactive experience and provide convenience for users to purchase the same clothing items.

[0168] The following describes the application of the video clothing detection method provided in this embodiment in a real-world scenario.

[0169] With the rapid development of the internet economy in recent years, the transaction volume of the online shopping market reached 13 trillion yuan in 2021, and the growth rate still maintains a strong momentum. Among them, the transaction volume of clothing accounts for as high as 57.5%. In the clothing online shopping scenario, celebrity-endorsed products and other star-chasing shopping are very popular. However, due to limitations such as image quality, special image angles, and object size, the image recognition capabilities of existing clothing retrieval algorithms under some special conditions are still insufficient to meet expectations.

[0170] Based on the aforementioned broad market prospects and current technical challenges, this disclosure provides an AI-powered celebrity-endorsed clothing shopping guide system based on video streams. This system integrates functions such as celebrity recognition, clothing attribute analysis, and clothing recommendation. It uses video streams to compensate for the shortcomings of single-frame images in clothing retrieval scenarios by using interval information, thereby improving the accuracy of clothing retrieval and recommendation and providing users with a more accurate and user-friendly clothing shopping guide system.

[0171] Please refer to Figure 8 This illustrates the business process of the apparel shopping guide system in an embodiment of this disclosure. For example... Figure 8 As shown, the process includes:

[0172] Step S801: Obtain the video to be processed and extract frames from the video to obtain multiple video frames.

[0173] The system can receive a video uploaded by a user through the video upload interface provided by the apparel shopping guide system, and use that video as the video to be processed.

[0174] In some embodiments, the video to be processed can be subjected to frame extraction at a preset frequency to obtain multiple video frames corresponding to the video to be processed. Then, subsequent steps S802, S803, and S804 can be performed based on these multiple video frames. Steps S802, S803, and S804 can be performed simultaneously, and the corresponding person (celebrity) recognition results, video segmentation results, and clothing attribute results can be obtained respectively.

[0175] Step S802: Identify the target person in each video frame to obtain the person identification result.

[0176] In some embodiments, facial features of people in video frames can be extracted to identify people in each video frame.

[0177] The process involves using face detection technology to obtain facial images from video frames, followed by using facial landmark detection technology to obtain the location information of five key points corresponding to the facial image. These five key points include the location of the left eye, right eye, nose, left corner of the mouth, and right corner of the mouth. A manually defined standard face is used as the correction benchmark for face alignment. The detected face is aligned with the standard face to obtain the aligned facial image. Then, a facial feature extractor is used to extract the local texture features of the facial image and reduce its dimensionality to obtain the facial features of the person in the video frame.

[0178] In some embodiments, in order to identify a target person in the video to be processed, such as a celebrity in the video to be processed, a preset face database corresponding to the target person (celebrity) can be established in advance. The method for constructing the preset face database includes: receiving an imported face sample image of the target person through a preset face interface, performing the face detection and face alignment process described above on the face sample image to obtain a partial face image corresponding to the target person.

[0179] The system can import multiple face sample images, and correspondingly, obtain multiple partial face images. To improve the accuracy of identifying the target person, the multiple partial face images can be cleaned to obtain a preset number of standard face images corresponding to the target person. These preset number of standard face images are used to verify the face image to be identified multiple times. For example, the preset number can be set to 7.

[0180] In some embodiments, the process of recognizing people in the current video frame is accomplished as follows: N standard face images corresponding to target people are extracted. With 7 standard face images corresponding to each target person, 7N standard face images are obtained. Feature extraction is performed on each standard face image to obtain 7N standard face features. With each standard face feature being a 2048-dimensional feature vector, a 7N×2048 feature matrix is ​​obtained. Simultaneously, feature extraction is performed on the face image corresponding to the person in the current video frame to obtain a 2048-dimensional feature vector. The distance between this feature vector and the 7N×2048 feature matrix is ​​calculated 7N times to obtain a 1×7N distance matrix. Here, 1 refers to the detection of 1 face in the current video frame. With the detection of M faces, an M×7N distance matrix is ​​obtained.

[0181] To facilitate understanding, we will take the detection of one face in the current video frame, resulting in a 1×7N distance matrix, as an example. Based on a preset distance threshold, this 1×7N distance matrix is ​​binarized: elements greater than the distance threshold are set to the first value, and elements less than the distance threshold are set to the second value, resulting in a 1×7N binary matrix to be verified. If all elements in this binary matrix are 0, it indicates that the person in the video frame is not a target person in the preset face database. If there is one 1 in the binary matrix, the target person corresponding to the element with the value of 1 is taken as the target person corresponding to the face in the current video frame. If there are at least two 1s in the binary matrix, it is determined whether there is only one target person corresponding to the element with the value of 1. If at least two elements with the value of 1 are located in the same target person's interval, then that same target person is taken as the target person corresponding to the face in the current video frame. If at least two elements with the value of 1 are located in the interval of at least two target persons, it is necessary to further determine one of the at least two target persons as the target person corresponding to the face in the current video frame. Since the vector distance between the same person is small and the vector distance between different people is large, for each of the at least two target persons, the feature distance between the 2048-dimensional feature vector corresponding to the video frame and the feature distance between the 7 standard facial features of each target person can be calculated. Then, at least two feature distance sums can be obtained, and the target person corresponding to the smallest feature distance sum is taken as the target person corresponding to the face in the current video frame.

[0182] Step S803: Divide multiple video frames into partitions to obtain video partitioning results.

[0183] The video partitioning result includes partitioning information for each video frame.

[0184] In some embodiments, for each video frame, the change category of the video frame can be determined based on the adjacent video frames corresponding to the video frame, and then multiple video frames can be divided based on the change category corresponding to each video frame to obtain the partition information corresponding to each video frame.

[0185] Specifically, based on the temporal relationship between multiple video frames, a preset number of video frames can be extracted from the multiple video frames using a sliding window of a preset length. These preset number of video frames are then input into a video segmentation network to determine the change category of the intermediate video frames. The video segmentation network can determine the change category of the intermediate video frames based on the changes in the content of the input preset number of video frames. This change category can include a first category (no change) and a second category (abrupt changes and gradual changes). After obtaining the change category of each video frame in the multiple video frames, the center point of N consecutive video frames of the second category is used as a segmentation benchmark. The obtained segmentation benchmark is then used to partition the multiple video frames, obtaining the partition information corresponding to each video frame.

[0186] For example, if there are 20 video frames, where the 1st to 8th video frames belong to the first category, the 9th to 11th video frames belong to the second category, the 12th to 14th video frames belong to the first category, the 15th video frame belongs to the second category, and the 16th to 20th video frames belong to the first category; then the 10th video frame in the 9th to 11th video frames of the second category can be used as the first dividing criterion, and the 15th video frame in the second category can be used as the second dividing criterion; then, these 20 video frames can be divided into the first partition of the 1st to 9th video frames, the second partition of the 11th to 14th video frames, and the third partition of the 16th to 20th video frames.

[0187] In some embodiments, the video segmentation network is a 3D convolutional network, which can have 5 or 6 layers. The input data of the 3D convolutional network is a 5D matrix in the format of batch-size × channel (number of channels in the video frame) × temporal-length (number of video frames) × height (height of the video frame) × width (width of the video frame). The output is a 1*3 logits. The loss function is multi-logits multi-class cross-entropy.

[0188] Step S804: Identify the clothing in each video frame to obtain the clothing attribute results corresponding to each video frame.

[0189] In some embodiments, the process of identifying clothing in each video frame may include a clothing detection step and a clothing attribute classification step.

[0190] In some embodiments, the garment testing step includes a two-stage testing method.

[0191] In the first stage, a candidate bounding box detection network is used to detect the positions of candidate bounding boxes for clothing in video frames. In this first stage, based on the feature map corresponding to the video frame, several anchor boxes are placed at each feature point in the feature map (anchor boxes of different sizes are placed with the feature point as the center). Each anchor box can predict the position of a candidate bounding box. During training, anchor boxes that are close to the actual clothing bounding box positions are used as positive samples, and others are used as negative samples. This makes the predicted bounding box positions corresponding to the anchor boxes of the above positive samples more accurate during testing. The top 100 predicted bounding boxes with the highest scores are selected as candidate bounding boxes.

[0192] In the second stage, the candidate boxes obtained in the first stage are subjected to region pooling on the feature map. The pooled features are used to classify the candidate boxes and further optimize them (determine the position offset and adjust the position of the candidate boxes), and finally obtain the category and final position of the clothing box.

[0193] In some embodiments, based on the final position of the clothing bounding box obtained in the clothing detection step, a clothing image corresponding to the clothing can be extracted from the video frame. To effectively extract clothing features, the clothing image can be scaled to 256×256 pixels and then input into a deep convolutional neural network to obtain a 2048-dimensional clothing feature vector. Based on the category of the clothing bounding box obtained in the clothing detection step, three classifiers corresponding to that category are selected. These three classifiers are used to classify the clothing category, texture, and main color of the clothing, respectively. These three classifiers share the previously extracted 2048-dimensional clothing feature vector. In each attribute classifier, a non-linear transformation is performed to convert the 2048-dimensional clothing feature vector into a 128-dimensional attribute feature vector, which is then subjected to a softmax operation to obtain the category of the corresponding attribute.

[0194] In some embodiments, the process of identifying clothing in each video frame may further include a similar clothing retrieval step. This similar clothing retrieval step may include a data augmentation process and a data retrieval process.

[0195] In some embodiments, this data augmentation process is used to intelligently augment the currently obtained clothing images. Since there are few samples of the same style of clothing input, and clothing often has characteristics such as wrinkles, deformations, and uneven size proportions, relying solely on the clothing images themselves for feature capture can be subject to randomness, chance, and background noise. Therefore, intelligent data augmentation of the clothing images is necessary to enhance the network's recognition of clothing details, such as texture patterns. Data augmentation methods include various affine transformations such as translation, scaling, and flipping. This allows the generation of the clothing image corresponding to the clothing in the video frame and at least one similar clothing image obtained through data augmentation.

[0196] In some embodiments, for the clothing in the video frame, a data retrieval process can be performed using the clothing image obtained through the data augmentation process and at least one similar clothing image. Compared to retrieving only the clothing image, this can improve retrieval accuracy. The data augmentation process can employ different retrieval strategies based on different data volumes, including a first retrieval strategy for tens of millions to hundreds of millions of data volumes and a second retrieval strategy for millions of data volumes.

[0197] Please see Figure 9A It illustrates a flowchart of a first retrieval strategy, which includes:

[0198] Step 1: Quantize and encode each preset clothing vector in the preset clothing library to obtain the corresponding quantization and encoding result.

[0199] Please see Figure 9A First, all preset clothing vectors 911 are coarsely clustered to obtain K coarse clustering results 912, and the cluster center corresponding to each clustering result is saved. For each clustering result, all preset clothing vectors 913 of that clustering result are uniformly divided into L feature segments 914. Clustering is performed on each feature segment 914 to obtain the cluster center of each feature segment. Based on the cluster center of each feature segment, each feature segment is quantized and encoded to obtain L quantized values ​​corresponding to each preset clothing vector. These L quantized values ​​are used as the quantization encoding result 915 corresponding to the preset clothing vector. Figure 9A In the first preset clothing vector, after quantization encoding, L quantization values ​​can be obtained: "14, 201, 34, 67".

[0200] Step 2: Based on the quantization encoding method in Step 1, the feature vectors of the clothing to be retrieved are encoded to obtain clothing codes. The clothing codes are then compared with the quantization encoding results corresponding to each preset clothing vector. The matching clothing data corresponding to the preset clothing vectors is used as the retrieval results.

[0201] In step two, for the clothing feature vector to be retrieved, the clothing feature vector is first compared with the cluster center corresponding to each clustering result to determine the most similar cluster center. Using the same method as in step one, the clothing feature vector is evenly divided into L feature segments. Based on the cluster center of each feature segment corresponding to the most similar cluster center, each feature segment is quantized and encoded to obtain L quantized values ​​corresponding to the clothing feature vector. Among the quantized encoding results corresponding to the multiple preset clothing vectors corresponding to the most similar cluster center, the multiple preset clothing vectors most similar to the L quantized values ​​corresponding to the clothing feature vector are determined as the preset clothing vectors for matching.

[0202] Please see Figure 9B It illustrates a flowchart of a second retrieval strategy, which includes:

[0203] Using a clothing image and at least one similar clothing image obtained through data augmentation as the query image, extract the overall feature vector and local feature vectors corresponding to the query image. Simultaneously, for each preset clothing image in the preset clothing database, extract the corresponding overall feature vector and local feature vectors using the same feature extraction method.

[0204] Please see Figure 9BIn the process of feature extraction for the query image 921 and the preset clothing image 931, a shared-weight backbone network can be used to extract features from the query image 921 and the preset clothing image 931 respectively. This backbone network can be a pyramid network, which can extract multi-layer features from the input image. Then, adaptive window pooling is performed on the multi-layer features corresponding to the query image 921 to obtain the overall feature vector 922 and local feature vector 923 for the query image 921; max pooling is performed on the multi-layer features corresponding to the preset clothing image 931 to obtain the overall feature vector 932 and local feature vector 933 for the preset clothing image 931. Next, the local feature similarity 941 between each pair of the local feature vectors 923 corresponding to the query image 921 and 933 corresponding to the preset clothing image 931 is determined; the overall similarity 942 between the overall feature vector 922 corresponding to the query image 921 and 932 corresponding to the preset clothing image 931 is also determined. Based on these local similarities 941 and overall feature similarities 942, a corresponding graph inference network is constructed. In this network, one node represents the overall feature similarity, and other nodes represent the local feature similarities of the query image 921 and the preset clothing image 931, respectively. Each edge represents the relationship between two similarities. Finally, the cross-entropy classification loss function is used to allow the network to judge the nodes of the global features to determine whether the query image 921 and the preset clothing image 931 belong to the same style of image, resulting in the output result 951.

[0205] In some embodiments, during the process of extracting multi-layer features using a pyramid network, a triplet loss function can be used to assist the network in discovering difficult samples. For each sample, the maximum distance between positive samples (i.e., the most difficult positive sample) and the minimum distance between negative samples (i.e., the most difficult negative sample) are taken as the optimization objective of the loss function, so that the distance between samples of the same class continuously decreases and the distance between samples of different classes continuously increases, resulting in a better feature space to ensure effective learning.

[0206] Step S805: Based on the clothing attribute results corresponding to each video frame and the partition information corresponding to each video frame, compare and vote on the clothing attribute results of video frames within the same partition to obtain the same clothing attribute results for video frames within the same partition.

[0207] In some embodiments, the comparison and voting process is used to cluster the clothing attribute results of each video frame within a partition, determine the appearance of the same item at different times, and then vote based on the clothing attribute results of the same object at different time dimensions to finally obtain the final clothing attribute result of the object. Because the same object in adjacent frames of the same scene in the video does not change much, all identified objects in different frames within the same partition can be aggregated into the same object appearing on the timeline.

[0208] In some embodiments, during the comparison and voting process, different voting weights can be assigned to each video frame within the same partition based on the video frame quality corresponding to different video frames. During the comparison and voting process, the final clothing attribute result of the object can be obtained based on the voting weights corresponding to different video frames. The video frame quality can include the brightness, sharpness, and occlusion degree of the clothing image.

[0209] Step S806: During the process of displaying the target video frame in the video to be processed, display the character information of the target person in the target video frame and the clothing attribute results corresponding to the target person.

[0210] In some embodiments, during the display of a target video frame from the video to be processed, the target video frame displays the target person's information and the corresponding clothing attribute results. The aforementioned person information and clothing attribute results can be displayed as rectangular boxes showing the location / range corresponding to the person and clothing, and the person's identity information and the clothing's category information can be displayed as text.

[0211] In some embodiments, please refer to Figure 10 , Figure 10 A schematic diagram of a clothing shopping guide interface is shown. The interface includes a video playback area 1001 for playing the video to be processed, using the display of a target video frame as an example. It also includes a character display area 1002 to display the character information of the target character in the target video frame, and an clothing display area 1003 to display the clothing attribute results corresponding to the target character in the target video frame.

[0212] In some embodiments, in response to a click operation on any clothing image in the clothing display area 1003, the clothing image corresponding to the click operation can be displayed in the clothing selection area 1004. At the same time, multiple similar clothing items corresponding to the clothing image and a purchase link for each similar clothing item are displayed in the clothing purchase area 1005. In response to a trigger operation on a target similar clothing item among the multiple similar clothing items, the user is redirected to the purchase interface corresponding to the target similar clothing item.

[0213] In some embodiments, after obtaining consistent voting results across intervals, video frames are smoothed to ensure that the identification information (person information and clothing attribute results) does not flicker continuously during frame skipping. The smoothed person information and clothing attribute results are displayed on the video frames as rectangles and classification information. Finally, the video is synthesized from the video frames, and the video and various SDK information are returned to the front-end page (clothing shopping guide interface). The results of the above functional steps are displayed on the front-end page accordingly.

[0214] Based on the above embodiments, compared with image retrieval and clothing recommendation based on single-frame images, this clothing shopping guide system can identify and recommend all clothing features within the uploaded video, not limited to a single-frame image, thus providing a wider range of recommendations. Simultaneously, image enhancement processing is added before feature retrieval, using various data enhancement methods such as translation, rotation, and scaling to strengthen the recognition of clothing details, improve the diversity of feature retrieval input, and enhance the retrieval effect. Furthermore, it provides algorithm result voting processing for video intervals, making the clothing retrieval results within the interval more stable, avoiding inconsistencies in retrieval results for the same clothing in a video due to changes in lighting, angle, etc., resulting in a better visual experience. This disclosure provides two single-frame image feature retrieval methods with data volumes of millions and hundreds of millions, allowing for the selection of the corresponding implementation method based on specific business scale and data volume.

[0215] Based on the foregoing embodiments, this disclosure provides a video clothing detection device. The device includes various units and modules included in each unit, which can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0216] Figure 11This is a schematic diagram of the composition structure of a video clothing detection device provided in an embodiment of this disclosure, as shown below. Figure 11 As shown, the video clothing detection device 1100 includes: a partitioning module 1101, a detection module 1102, and a calibration module 1103, wherein:

[0217] The partitioning module 1101 is used to partition multiple video frames of the video to be processed and determine the partitioning information corresponding to each video frame.

[0218] The detection module 1102 is used to detect each of the video frames and obtain the detection result corresponding to the target person in each video frame; the detection result includes the person information of the target person in the video frame and the clothing information corresponding to the target person;

[0219] The correction module 1103 is used to correct the detection results of video frames within the same partition based on the partition information corresponding to each video frame, so as to obtain the target detection result corresponding to each video frame; wherein, the target detection results corresponding to video frames within the same partition are the same for the detection results of the target person.

[0220] In some embodiments, the partitioning module 1101 is further configured to:

[0221] Obtain multiple video frames corresponding to the video to be processed;

[0222] Based on the set of adjacent video frames corresponding to each video frame, a change category is determined for each video frame; the change category is used to characterize the degree of change of the video frame relative to its corresponding adjacent video frames.

[0223] Based on the change category corresponding to each video frame, the multiple video frames are divided to obtain the partition information corresponding to each video frame.

[0224] In some embodiments, the partitioning module 1101 is further configured to:

[0225] For each video frame, at least one video frame adjacent to the video frame is taken as the set of adjacent video frames corresponding to the video frame.

[0226] The set of adjacent video frames corresponding to the video frame is input into the trained video segmentation network to obtain the change category corresponding to the video frame.

[0227] In some embodiments, the change category includes a first category representing a first degree of change and a second category representing a second degree of change; the second degree of change is higher than the first degree of change; the partitioning module 1101 is further configured to:

[0228] Based on the temporal relationship of the multiple video frames, the video frames corresponding to each change category are clustered to obtain at least one set of video frames; the set of video frames includes one video frame or at least two consecutive video frames, and the at least two consecutive video frames have the same change category.

[0229] Using the video frame located at the center of the target video frame set as the dividing benchmark, the multiple video frames are divided to obtain the partition information corresponding to each video frame; the target video frame set is the video frame set of the second category.

[0230] In some embodiments, the detection module 1102 is further configured to:

[0231] For each video frame, perform person detection on the video frame to determine the person information corresponding to the target person in the video frame;

[0232] Clothing detection is performed on the video frames to determine the clothing information corresponding to the target person.

[0233] In some embodiments, the detection module 1102 is further configured to:

[0234] Perform face detection on the video frame to determine the face image corresponding to the face in the video frame;

[0235] Facial features are extracted from the face image to obtain the face features to be matched corresponding to the face;

[0236] Obtain a preset facial feature database, which includes at least one preset person and person information and multiple facial features corresponding to each preset person;

[0237] Based on the feature distance between the face feature to be matched and the multiple face features corresponding to each preset person, the target preset person corresponding to the face is determined among the multiple preset people, and the person information corresponding to the target preset person is determined as the person information corresponding to the target person.

[0238] In some embodiments, the detection module 1102 is further configured to:

[0239] For each of the preset characters, the feature distance between the face feature to be matched and each face feature corresponding to the preset character is determined respectively;

[0240] By comparing the feature distance of each facial feature with a preset distance threshold, a similarity result corresponding to each facial feature is determined; the similarity result is used to characterize whether the face corresponds to a preset person corresponding to the facial feature;

[0241] Based on the similarity result corresponding to each facial feature, the target preset person corresponding to the face is determined among the plurality of preset persons.

[0242] In some embodiments, the detection module 1102 is further configured to:

[0243] If all the first similarity results correspond to a pre-matched person, the pre-matched person is determined as the target person; the first similarity result indicates that the face corresponds to the pre-matched person whose facial features are represented.

[0244] In the case where the first similarity result corresponds to at least two matched preset persons, the target preset person is determined from the at least two matched preset persons based on the feature distance between the face feature to be matched and the sum of the feature distances between the face feature to be matched and the multiple face features corresponding to each of the matched preset persons.

[0245] In some embodiments, the clothing information includes the clothing category, and the detection module 1102 is further configured to:

[0246] Extract the clothing feature map corresponding to the video frame;

[0247] Based on the clothing feature map, the detection box information corresponding to the clothing in the video frame is determined;

[0248] Based on the detection box information corresponding to the clothing, the clothing features corresponding to the clothing are determined;

[0249] Based on the clothing characteristics corresponding to the clothing, the clothing category of the clothing is determined.

[0250] In some embodiments, the detection module 1102 is further configured to:

[0251] Based on the detection box information corresponding to the clothing, the clothing image corresponding to the clothing is extracted from the video frame;

[0252] Feature extraction is performed on the clothing image to obtain the clothing features corresponding to the clothing.

[0253] In some embodiments, the clothing features include multiple clothing sub-features; the detection module 1102 is further configured to:

[0254] Based on the detection box information corresponding to the clothing, the clothing image corresponding to the clothing is extracted from the video frame;

[0255] The clothing image is subjected to data augmentation processing to obtain at least one similar clothing image corresponding to the clothing;

[0256] Feature extraction is performed on the clothing image and the at least one similar clothing image to obtain multiple clothing sub-features corresponding to the clothing.

[0257] In some embodiments, the clothing information includes data information of the clothing; the detection module 1102 is further configured to:

[0258] The target retrieval strategy is determined based on the amount of data in the preset clothing database between the first retrieval strategy and the second retrieval strategy; the amount of data corresponding to the first retrieval strategy is higher than the amount of data corresponding to the second retrieval strategy; the preset clothing database includes multiple preset clothing items and data information corresponding to each preset clothing item;

[0259] Using the target retrieval strategy, at least one preset garment that matches the garment is determined from the plurality of preset garments;

[0260] The data information corresponding to each preset garment that matches the garment is determined as the data information of the garment.

[0261] In some embodiments, when the target retrieval strategy is the first retrieval strategy, the detection module 1102 is further configured to:

[0262] Based on the clothing image, determine the clothing feature vector corresponding to the clothing image;

[0263] Among the multiple first central features corresponding to the multiple preset clothing items, a target first central feature that matches the clothing feature vector is determined; the multiple first central features are determined by clustering multiple preset clothing vectors in the preset clothing library, and each first clustering result corresponds to a first central feature;

[0264] Based on the first central feature of the target, the clothing feature vector is quantized and encoded to obtain the clothing code;

[0265] The preset clothing corresponding to the preset clothing code corresponding to the target encoding result is determined as at least one preset clothing that matches the clothing; the target encoding result is at least one quantization encoding result that matches the clothing code among multiple quantization encoding results corresponding to the target first central feature; the method for quantizing the preset clothing vector corresponding to the target first central feature is the same as the method for quantizing the clothing feature vector.

[0266] In some embodiments, the detection module 1102 is further configured to:

[0267] Based on different feature locations, the clothing feature vector is split into features within each feature to obtain a clothing sub-vector corresponding to each feature location of the clothing feature vector.

[0268] Based on the second central feature of each feature position corresponding to the first central feature of the target, the clothing sub-vector corresponding to each feature position is quantized and encoded to obtain the quantized value of the clothing sub-vector corresponding to each feature position; the second central feature is determined by clustering the preset sub-vectors of multiple feature positions corresponding to the first central feature of the target, and each second clustering result obtained by clustering corresponds to a second central feature.

[0269] The clothing code is determined based on the quantization value of the clothing sub-vector corresponding to each of the aforementioned feature positions.

[0270] In some embodiments, when the target retrieval strategy is the second retrieval strategy, the detection module 1102 is further configured to:

[0271] Based on the clothing image, determine the overall feature vector and at least one local feature vector corresponding to the clothing image;

[0272] For each of the plurality of preset garments, a first similarity is determined by determining the preset overall vector and the overall feature vector corresponding to the preset garment, and a second similarity is determined between each preset local vector and each local feature vector corresponding to the preset garment; based on the first similarity and at least one second similarity, a preset similarity between the preset garment and the garment is determined.

[0273] Based on the preset similarity corresponding to each preset garment, at least one preset garment that matches the garment is determined from the plurality of preset garments.

[0274] In some embodiments, the clothing information of the target person includes detection box information and clothing category corresponding to each garment in the video frame; the detection module 1102 is further configured to:

[0275] For each partition, each garment in each video frame within the partition is classified to obtain at least one detection category; the detection box information of the garment corresponding to the detection category in the corresponding at least one video frame to be corrected satisfies a preset overlap condition.

[0276] For each garment, the target garment category is determined based on the garment category corresponding to the garment in each video frame to be corrected.

[0277] In some embodiments, the detection module 1102 is further configured to:

[0278] Obtain the video frame quality corresponding to each of the video frames to be corrected;

[0279] Based on the video frame quality corresponding to each video frame to be corrected, a voting weight is determined for each video frame to be corrected; the voting weight corresponding to the video frame to be corrected is positively correlated with the video frame quality corresponding to the video frame to be corrected.

[0280] Based on the voting weight corresponding to each of the video frames to be corrected and the clothing category corresponding to the clothing in each of the video frames to be corrected, the target clothing category corresponding to the clothing is determined.

[0281] In some embodiments, the detection module 1102 is further configured to:

[0282] For each video frame to be corrected, the clothing area corresponding to the video frame to be corrected is determined based on the detection box information corresponding to the clothing.

[0283] Based on the video frame to be corrected and the clothing area, determine the video frame quality corresponding to the video frame to be corrected;

[0284] The video frame quality includes at least one of the following: the degree of occlusion by the clothing, the clarity of the clothing area corresponding to the clothing, and the brightness of the clothing area corresponding to the clothing.

[0285] In some embodiments, the video frame quality includes quality information of at least one quality dimension, and the clothing category includes at least one clothing subcategory; the detection module 1102 is further configured to:

[0286] For each of the aforementioned clothing subcategories, based on the degree of correlation between the clothing subcategory and each of the aforementioned quality dimensions, and the quality information of each of the aforementioned quality dimensions, the voting subweight corresponding to the clothing subcategory is determined;

[0287] The step of determining the target clothing category corresponding to the clothing based on the voting weight corresponding to each of the video frames to be corrected and the clothing category corresponding to the clothing in each of the video frames to be corrected includes:

[0288] For each clothing sub-category, the target clothing sub-category is determined based on the voting sub-weight corresponding to the clothing sub-category and the clothing sub-category corresponding to the clothing in each video frame to be corrected.

[0289] In some embodiments, the video clothing detection device 1100 further includes a display module.

[0290] The display module is used to play the video to be processed through the clothing display interface; during the process of displaying the target video frame in the video to be processed, the character information of the target person in the target video frame and the clothing information corresponding to the target person are displayed.

[0291] In some embodiments, the display module is further configured to:

[0292] The character information of the target character in the target video frame is displayed through the character display area in the clothing display interface;

[0293] The clothing display area in the clothing display interface displays the clothing information corresponding to the target character in the target video frame; the clothing information includes a partial clothing image and clothing category corresponding to each piece of clothing in the target video frame.

[0294] In some embodiments, the display module is further configured to:

[0295] Receive a trigger operation for the target clothing image in the local clothing image corresponding to each of the aforementioned clothing items;

[0296] In response to the triggering operation, at least one related garment and a purchase link corresponding to each related garment are displayed in the garment purchase area of ​​the garment display interface.

[0297] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this disclosure can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0298] It should be noted that, in the embodiments of this disclosure, if the above-described video clothing detection method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this disclosure, or the part that contributes to related technologies, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk. Thus, the embodiments of this disclosure are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0299] This disclosure provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0300] This disclosure provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium may be transient or non-transient.

[0301] This disclosure provides a computer program including computer-readable code, wherein when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0302] This disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0303] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referenced interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0304] Figure 12 This is a schematic diagram of the hardware entity of a video clothing detection device provided in an embodiment of this disclosure, such as... Figure 12 As shown, the hardware entity of the video clothing detection device 1200 includes a processor 1201 and a memory 1202, wherein the memory 1202 stores a computer program that can run on the processor 1201, and the processor 1201 executes the program to implement the steps in the method of any of the above embodiments.

[0305] The memory 1202 stores computer programs that can run on the processor. The memory 1202 is configured to store instructions and applications that can be executed by the processor 1201. It can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 1201 and the various modules in the video clothing detection device 1200. It can be implemented by flash memory or random access memory (RAM).

[0306] When the processor 1201 executes the program, it implements the steps of the video clothing detection method described above. The processor 1201 typically controls the overall operation of the video clothing detection device 1200.

[0307] This disclosure provides a computer storage medium storing one or more programs that can be executed by one or more processors to implement the steps of the video clothing detection method as described in any of the above embodiments.

[0308] It should be noted that the descriptions of the storage medium and device embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0309] The aforementioned processor can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that other electronic devices can also implement the functions of the aforementioned processor, and this disclosure does not specifically limit the specific implementation.

[0310] The aforementioned computer storage media / memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; or it can be various terminals that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0311] It should be understood that the phrase "an embodiment" or "one embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this disclosure. Therefore, "in one embodiment" or "one embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this disclosure, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure. The sequence numbers of the above embodiments of this disclosure are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0312] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0313] In the several embodiments provided in this disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0314] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0315] In addition, each functional unit in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0316] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0317] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0318] The above description is merely an embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video clothing detection method, characterized in that, The method includes: The video to be processed is divided into multiple video frames, and the partition information corresponding to each video frame is determined; the partition information is used to determine the partition in which the video frame is located, and the video frames within the partition are consecutive. Each video frame is detected to obtain a detection result corresponding to the target person in each video frame; the detection result includes the person information of the target person in the video frame and the clothing information corresponding to the target person, and the clothing information of the target person includes the detection box information and clothing category corresponding to each garment in the video frame; Based on the partition information corresponding to each video frame, the detection results of video frames within the same partition are corrected to obtain the target detection result corresponding to each video frame; wherein, the target detection results corresponding to video frames within the same partition are the same for the detection results of the target person. The step of correcting the detection results of video frames within the same partition based on the partition information corresponding to each video frame to obtain the target detection result corresponding to each video frame includes: For each partition, each garment in each video frame within the partition is classified to obtain at least one detection category; the detection box information of the garment corresponding to the detection category in the corresponding at least one video frame to be corrected satisfies a preset overlap condition. For each garment, the target garment category is determined based on the garment category corresponding to the garment in each video frame to be corrected.

2. The method according to claim 1, characterized in that, The step of partitioning multiple video frames of the video to be processed and determining the partition information corresponding to each video frame includes: Obtain multiple video frames corresponding to the video to be processed; Based on the set of adjacent video frames corresponding to each video frame, a change category is determined for each video frame; the change category is used to characterize the degree of change of the video frame relative to its corresponding adjacent video frames. Based on the change category corresponding to each video frame, the multiple video frames are divided to obtain the partition information corresponding to each video frame.

3. The method according to claim 2, characterized in that, The step of determining the change category corresponding to each video frame based on the set of adjacent video frames corresponding to each video frame includes: for each video frame, taking at least one video frame adjacent to the video frame as the set of adjacent video frames corresponding to the video frame; inputting the set of adjacent video frames corresponding to the video frame into the trained video segmentation network to obtain the change category corresponding to the video frame. And / or, The change categories include a first category representing a first degree of change and a second category representing a second degree of change; the second degree of change is higher than the first degree of change; the step of dividing the plurality of video frames based on the change category corresponding to each video frame to obtain partition information corresponding to each video frame includes: clustering the video frames corresponding to each change category based on the temporal relationship corresponding to the plurality of video frames to obtain at least one set of video frames; the set of video frames includes one video frame or at least two consecutive video frames, and the at least two consecutive video frames have the same change category; using the video frame located at the center of the target set of video frames as the dividing benchmark, dividing the plurality of video frames to obtain partition information corresponding to each video frame; the target set of video frames is the set of video frames of the second category.

4. The method according to any one of claims 1 to 3, characterized in that, The step of detecting each video frame to obtain the detection result corresponding to the target person in each video frame includes: For each video frame, perform person detection on the video frame to determine the person information corresponding to the target person in the video frame; Clothing detection is performed on the video frames to determine the clothing information corresponding to the target person.

5. The method according to claim 4, characterized in that, The step of performing person detection on the video frame to determine the person information corresponding to the target person in the video frame includes: Perform face detection on the video frame to determine the face image corresponding to the face in the video frame; Facial features are extracted from the face image to obtain the face features to be matched corresponding to the face; Obtain a preset facial feature database, which includes multiple preset individuals and their corresponding information and multiple facial features; Based on the feature distance between the face feature to be matched and the multiple face features corresponding to each preset person, the target preset person corresponding to the face is determined among the multiple preset people, and the person information corresponding to the target preset person is determined as the person information corresponding to the target person.

6. The method according to claim 5, characterized in that, The step of determining the target preset person corresponding to the face from among the multiple preset persons based on the feature distance between the face feature to be matched and the multiple face features corresponding to each preset person includes: For each of the preset characters, the feature distance between the face feature to be matched and each face feature corresponding to the preset character is determined respectively; By comparing the feature distance of each facial feature with a preset distance threshold, a similarity result corresponding to each facial feature is determined; the similarity result is used to characterize whether the face corresponds to a preset person corresponding to the facial feature; Based on the similarity result corresponding to each facial feature, the target preset person corresponding to the face is determined among the plurality of preset persons.

7. The method according to claim 6, characterized in that, The step of determining the target preset person corresponding to the face among the plurality of preset persons based on the similarity result corresponding to each of the facial features includes: If all the first similarity results correspond to a pre-matched person, the pre-matched person is determined as the target person; the first similarity result indicates that the face corresponds to the pre-matched person whose facial features are corresponding to it. In the case where the first similarity result corresponds to at least two matched preset persons, the target preset person is determined from the at least two matched preset persons based on the feature distance between the face feature to be matched and the sum of the feature distances between the face feature to be matched and the multiple face features corresponding to each of the matched preset persons.

8. The method according to any one of claims 5 to 7, characterized in that, The clothing information includes the clothing category. The step of detecting clothing in the video frames to determine the clothing information corresponding to the target person includes: Extract the clothing feature map corresponding to the video frame; Based on the clothing feature map, the detection box information corresponding to the clothing in the video frame is determined; Based on the detection box information corresponding to the clothing, the clothing features corresponding to the clothing are determined; Based on the clothing characteristics corresponding to the clothing, the clothing category of the clothing is determined.

9. The method according to claim 8, characterized in that, The step of determining the clothing features corresponding to the clothing based on the detection box information corresponding to the clothing includes: extracting the clothing image corresponding to the clothing from the video frame based on the detection box information corresponding to the clothing; and extracting features from the clothing image to obtain the clothing features corresponding to the clothing. The clothing features include multiple clothing sub-features; determining the clothing features corresponding to the clothing based on the detection box information corresponding to the clothing includes: extracting the clothing image corresponding to the clothing from the video frame based on the detection box information corresponding to the clothing; performing data augmentation processing on the clothing image to obtain at least one similar clothing image corresponding to the clothing; and performing feature extraction on the clothing image and the at least one similar clothing image to obtain multiple clothing sub-features corresponding to the clothing. The clothing information includes the data information of the clothing; the step of detecting clothing in the video frame to determine the clothing information of the target person further includes: determining a target retrieval strategy based on the data volume of a preset clothing library in a first retrieval strategy and a second retrieval strategy; the data volume corresponding to the first retrieval strategy is higher than the data volume corresponding to the second retrieval strategy; the preset clothing library includes multiple preset clothing items and data information corresponding to each preset clothing item; using the target retrieval strategy, determining at least one preset clothing item that matches the clothing item among the multiple preset clothing items; and determining the data information corresponding to each preset clothing item that matches the clothing item as the data information of the clothing item.

10. The method according to claim 9, characterized in that, When the target retrieval strategy is the first retrieval strategy, the step of using the target retrieval strategy to determine at least one preset garment that matches the garment from the plurality of preset garments includes: Based on the clothing image, determine the clothing feature vector corresponding to the clothing image; Among the multiple first central features corresponding to the multiple preset clothing items, a target first central feature that matches the clothing feature vector is determined; the multiple first central features are determined by clustering multiple preset clothing vectors in the preset clothing library, and each first clustering result corresponds to a first central feature; Based on the first central feature of the target, the clothing feature vector is quantized and encoded to obtain the clothing code; The preset clothing corresponding to the preset clothing code corresponding to the target encoding result is determined as at least one preset clothing that matches the clothing; the target encoding result is at least one quantization encoding result that matches the clothing code among multiple quantization encoding results corresponding to the target first central feature; the method for quantizing the preset clothing vector corresponding to the target first central feature is the same as the method for quantizing the clothing feature vector.

11. The method according to claim 10, characterized in that, The step of quantizing and encoding the clothing feature vector based on the target first central feature to obtain the clothing code includes: Based on different feature locations, the clothing feature vector is split into features within each feature to obtain a clothing sub-vector corresponding to each feature location of the clothing feature vector. Based on the second central feature of each feature position corresponding to the first central feature of the target, the clothing sub-vector corresponding to each feature position is quantized and encoded to obtain the quantized value of the clothing sub-vector corresponding to each feature position; the second central feature is determined by clustering the preset sub-vectors of multiple feature positions corresponding to the first central feature of the target, and each second clustering result obtained by clustering corresponds to a second central feature. The clothing code is determined based on the quantization value of the clothing sub-vector corresponding to each of the aforementioned feature positions.

12. The method according to any one of claims 9 to 11, characterized in that, When the target retrieval strategy is the second retrieval strategy, the step of using the target retrieval strategy to determine at least one preset garment that matches the garment from the plurality of preset garments includes: Based on the clothing image, determine the overall feature vector and at least one local feature vector corresponding to the clothing image; For each of the plurality of preset garments, a first similarity is determined by determining the preset overall vector and the overall feature vector corresponding to the preset garment, and a second similarity is determined between each preset local vector and each local feature vector corresponding to the preset garment; based on the first similarity and at least one second similarity, a preset similarity between the preset garment and the garment is determined. Based on the preset similarity corresponding to each preset garment, at least one preset garment that matches the garment is determined from the plurality of preset garments.

13. The method according to claim 1, characterized in that, The step of determining the target clothing category corresponding to the clothing based on the clothing category corresponding to the clothing in each of the video frames to be corrected includes: Obtain the video frame quality corresponding to each of the video frames to be corrected; Based on the video frame quality corresponding to each video frame to be corrected, a voting weight corresponding to each video frame to be corrected is determined; the voting weight corresponding to the video frame to be corrected is positively correlated with the video frame quality corresponding to the video frame to be corrected. Based on the voting weight corresponding to each of the video frames to be corrected and the clothing category corresponding to the clothing in each of the video frames to be corrected, the target clothing category corresponding to the clothing is determined.

14. The method according to claim 13, characterized in that, The step of obtaining the video frame quality corresponding to each of the video frames to be corrected includes: For each video frame to be corrected, the clothing area corresponding to the video frame to be corrected is determined based on the detection box information corresponding to the clothing. Based on the video frame to be corrected and the clothing area, determine the video frame quality corresponding to the video frame to be corrected; The video frame quality includes at least one of the following: the degree of occlusion by the clothing, the clarity of the clothing area corresponding to the clothing, and the brightness of the clothing area corresponding to the clothing.

15. The method according to claim 13 or 14, characterized in that, The video frame quality includes quality information of at least one quality dimension, and the clothing category includes at least one clothing subcategory; determining the voting weight corresponding to each video frame to be corrected based on the video frame quality of each video frame to be corrected includes: For each of the aforementioned clothing subcategories, based on the degree of correlation between the clothing subcategory and each of the aforementioned quality dimensions, and the quality information of each of the aforementioned quality dimensions, the voting subweight corresponding to the clothing subcategory is determined; The step of determining the target clothing category corresponding to the clothing based on the voting weight corresponding to each of the video frames to be corrected and the clothing category corresponding to the clothing in each of the video frames to be corrected includes: For each clothing sub-category, the target clothing sub-category is determined based on the voting sub-weight corresponding to the clothing sub-category and the clothing sub-category corresponding to the clothing in each video frame to be corrected.

16. The method according to any one of claims 1 to 3, 5 to 7, 9 to 11, and 13 to 14, characterized in that, The method further includes: The video to be processed is played through the clothing display interface; During the process of displaying the target video frame in the video to be processed, the character information of the target person in the target video frame and the corresponding clothing information of the target person are displayed.

17. The method according to claim 16, characterized in that, The process of displaying the character information and clothing information of the target person in the target video frame includes: The character information of the target character in the target video frame is displayed through the character display area in the clothing display interface; The clothing display area in the clothing display interface displays the clothing information corresponding to the target character in the target video frame; the clothing information includes a partial clothing image and clothing category corresponding to each piece of clothing in the target video frame.

18. A video-based clothing detection device, characterized in that, include: The partitioning module is used to partition multiple video frames of the video to be processed and determine the partitioning information corresponding to each video frame. The partition information is used to determine the partition in which the video frame is located, and the video frames within the partition are continuous. The detection module is used to detect each of the video frames and obtain the detection result corresponding to the target person in each video frame; the detection result includes the person information of the target person in the video frame and the clothing information corresponding to the target person, and the clothing information of the target person includes the detection box information and clothing category corresponding to each piece of clothing in the video frame. The correction module is used to correct the detection results of video frames within the same partition based on the partition information corresponding to each video frame, so as to obtain the target detection result corresponding to each video frame; wherein, the target detection results corresponding to video frames within the same partition are the same for the detection results of the target person. The correction module is also used to classify each garment in each video frame within each partition to obtain at least one detection category; the detection box information of the garment corresponding to the detection category in the corresponding at least one video frame to be corrected satisfies a preset overlap condition. For each garment, the target garment category is determined based on the garment category corresponding to the garment in each video frame to be corrected.

19. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 17.

Images