Clothing identification method, device, equipment and storage medium

Abstract

The embodiment of the present invention provides a clothing recognition method, device, equipment and storage medium, the method comprising: extracting key frames at multiple different moments from a video to be recognized; detecting clothing areas and human body areas on the key frames; in response to detecting at least one clothing area and at least one human body area in the key frames, selecting a representative clothing area from the at least one clothing area, the representative clothing area meeting the following conditions: the overlap ratio between the representative clothing area and the human body area reaches a first threshold, the representative clothing area is a target clothing posture, and the clothing exposure ratio reaches a second threshold under the target clothing posture; clothing recognition is performed based on the selected representative clothing area. In this way, the recognition errors that may be caused by unstable clothing postures and incomplete exposure of noise clothing areas are reduced, thereby improving the accuracy of clothing recognition.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a clothing recognition method, device, equipment and storage medium. Background Art

[0002] Clothing is an indispensable item in people's daily life. With the emergence of online shopping platforms, people often buy clothing on online shopping platforms, which greatly facilitates people's lives. In the related technology, in order to further provide convenience for people, clothing that appears in the video can be identified by clothing recognition technology to provide people with relevant clothing information. The existing clothing recognition technology is still in the development stage, and the recognition accuracy is still low, which is difficult to meet the recognition requirements of actual application scenarios. For example, the maximum accuracy of most clothing recognition technologies based on open source data sets is less than 70%, which is difficult to meet the recognition requirements of the actual application scenarios with an accuracy higher than 90%. In particular, the following recognition problems are prone to occur: Since clothing is a non-rigid object, the difference in the posture of the clothing of the characters is easy to cause the occlusion and incomplete exposure of the clothing area, which may cause the same clothes appearing at two different times, t1 and t2, to be recognized as different clothes, or two different clothes appearing at two different times, t3 and t4, to be recognized as the same clothes. This is a serious challenge for clothing recognition in videos. Summary of the invention

[0003] The purpose of the embodiments of the present invention is to provide a clothing recognition method, device, equipment and storage medium to improve the accuracy of clothing recognition. The specific technical solution is as follows:

[0004] In a first aspect of the present invention, a clothing recognition method is provided, comprising:

[0005] Extract key frames at multiple different moments from the video to be identified;

[0006] Performing clothing area and human body area detection on the key frame;

[0007] In response to detecting at least one clothing region and at least one human body region in the key frame, selecting a representative clothing region from the at least one clothing region, wherein the representative clothing region satisfies the following conditions: an overlap ratio between the representative clothing region and the human body region reaches a first threshold, the representative clothing region is in a target clothing posture, and a clothing exposure ratio reaches a second threshold under the target clothing posture;

[0008] Clothing recognition is performed based on the selected representative clothing area.

[0009] In another aspect of the present invention, a clothing recognition device is provided, comprising:

[0010] A key frame extraction module is used to extract key frames at multiple different moments from the video to be identified;

[0011] A region detection module, used for detecting clothing regions and human body regions on the key frames;

[0012] a representative selection module, configured to select a representative clothing region from the at least one clothing region in response to detecting at least one clothing region and at least one human body region in the key frame, wherein the representative clothing region satisfies the following conditions: an overlap ratio between the representative clothing region and the human body region reaches a first threshold, the representative clothing region is in a target clothing posture, and a clothing exposure ratio reaches a second threshold under the target clothing posture;

[0013] The clothing recognition module is used to perform clothing recognition based on the selected representative clothing area.

[0014] In another aspect of the present invention, there is provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0015] Memory, used to store computer programs;

[0016] The processor is used to implement any of the above-mentioned steps of the clothing recognition method when executing the program stored in the memory.

[0017] In another aspect of the present invention, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes any of the above-mentioned clothing recognition methods.

[0018] In another aspect of the present invention, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the clothing recognition methods described above.

[0019] The clothing recognition method, apparatus, device and storage medium provided by the embodiments of the present invention perform clothing area and human body area detection on multiple key frames at different times in a video to be recognized, and in response to detecting at least one clothing area and at least one human body area, select a representative clothing area from the at least one clothing area, the overlapping ratio of the representative clothing area with the human body area reaches a first threshold, and the clothing exposure ratio reaches a second threshold under the target clothing posture, that is, the clothing area worn on the human body that meets the clothing posture requirements and is relatively completely exposed under the required clothing posture is used as the representative clothing area for clothing recognition, thereby reducing recognition errors that may be caused by unstable clothing postures and incomplete exposure of noise clothing areas, thereby improving the accuracy of clothing recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0021] Figure 1 FIG. 4 is an exemplary system architecture diagram in an embodiment of the present invention.

[0022] Figure 2 The figure is a flowchart of an exemplary clothing recognition method in an embodiment of the present invention.

[0023] Figure 3 The figure is a flowchart of an exemplary clothing recognition method in an embodiment of the present invention.

[0024] Figure 4 The figure is a flowchart of an exemplary clothing recognition method in an embodiment of the present invention.

[0025] Figure 5 The figure is a flowchart of an exemplary clothing recognition method in an embodiment of the present invention.

[0026] Figure 6 The figure is a flowchart of an exemplary clothing recognition method in an embodiment of the present invention.

[0027] Figure 7 The figure is a flowchart of an exemplary clothing recognition method in an embodiment of the present invention.

[0028] Figure 8 Schematic diagram of the structure of an exemplary clothing recognition device in an embodiment of the present invention.

[0029] Fig. 9 It is a schematic diagram of the structure of an exemplary electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0031] In order to solve the problem of cross-time style ambiguity in clothing recognition in videos, an embodiment of the present invention provides a clothing recognition method, which selects a representative clothing area in combination with a human body area, wherein the overlap ratio between the representative clothing area and the human body area reaches a first threshold, and the representative clothing area is a target clothing posture and the clothing exposure ratio reaches a second threshold under the target clothing posture. That is, clothing areas on the human body that meet the target clothing posture and are relatively completely exposed under the target clothing posture are used as representatives for clothing recognition, thereby reducing recognition errors that may be caused by unstable clothing postures and incompletely exposed noisy clothing areas. The clothing recognition method provided by the embodiment of the present invention can be executed by a server, or by a user terminal, etc. Taking the execution of the server as an example, Figure 1 This is a system architecture diagram in an embodiment of the present invention. The system includes a server and a user terminal. In practical applications, the server can execute the clothing recognition method provided in the embodiment of the present invention, which can be applied to processing scenarios such as clothing recognition or clothing recommendation, and send the processing results to the user terminal, so that the user of the user terminal can understand the clothing information based on this. The solution provided in the embodiment of the present invention is described in more detail below.

[0032] Figure 2 FIG. 1 is a flowchart of an exemplary clothing recognition method provided by an embodiment of the present invention. Figure 2 As shown, the clothing recognition method provided in this embodiment may at least include the following steps:

[0033] Step 201: extract multiple key frames at different moments from the video to be identified.

[0034] Step 202: Detect clothing areas and human body areas on the key frames.

[0035] Step 203, in response to at least one clothing area and at least one human body area being detected in the key frame, a representative clothing area is selected from the at least one clothing area, and the representative clothing area satisfies the following conditions: an overlap ratio between the representative clothing area and the human body area reaches a first threshold, the representative clothing area is in a target clothing posture, and a clothing exposure ratio reaches a second threshold under the target clothing posture.

[0036] Step 204: Perform clothing recognition based on the selected representative clothing area.

[0037] The video to be identified is a video for clothing identification.

[0038] The clothing exposure ratio is the ratio of clothing contained in the clothing area that is exposed in the key frame. The clothing exposure ratio can reflect the completeness of clothing exposure. The more complete the clothing exposure is, the higher the accuracy of clothing recognition is.

[0039] After extracting key frames from the video to be identified, clothing areas and human body areas are detected for each key frame. For a key frame, if at least one clothing area and at least one human body area are detected in the key frame, it means that there is clothing worn on the human body. At this time, a representative clothing area can be selected from the at least one detected clothing area, and the overlapping ratio of the representative clothing area and the human body area reaches a first threshold, that is, the clothing representing the clothing area is clothing worn on the human body, and the representative clothing area is the target clothing posture, that is, the clothing posture of the representative clothing area meets the clothing posture requirement, and the clothing exposure ratio reaches the second threshold corresponding to the required clothing posture, that is, the clothing is exposed more completely. In this way, the clothing area worn on the human body that meets the clothing posture requirement and is exposed more completely under the required clothing posture can be used as a representative for clothing recognition, thereby reducing recognition errors that may be caused by unstable clothing postures and incompletely exposed noise clothing areas.

[0040] Among them, the clothing posture is the posture of clothing when it is displayed, and the target clothing posture is the clothing posture that the representative clothing area is required to meet.

[0041] Based on this, in this embodiment, clothing area and human body area detection are performed on multiple key frames at different moments in the video to be identified. In response to detecting at least one clothing area and at least one human body area, a representative clothing area is selected from the at least one clothing area, and the overlapping ratio of the representative clothing area with the human body area reaches a first threshold, and the clothing exposure ratio reaches a second threshold under the target clothing posture. That is, the clothing area worn on the human body that meets the clothing posture requirements and is relatively completely exposed under the required clothing posture is used as the representative clothing area for clothing identification, thereby reducing the identification errors that may be caused by unstable clothing postures and incomplete exposure of noise clothing areas, thereby improving the accuracy of clothing recognition.

[0042] In one embodiment, the human body region includes a key human posture node at at least one position, and a representative clothing region is selected from at least one clothing region, such as Figure 3 As shown, the specific implementation method may include:

[0043] Step 301: Compare the clothing area in the same key frame with each human body area to obtain a first comparison result, and determine a first corresponding relationship based on the first comparison result, wherein the first corresponding relationship includes human body areas and clothing areas whose overlap ratio reaches a first threshold.

[0044] Step 302: for each human body region in the first corresponding relationship, compare the human body region with a preset human body posture key node set to obtain a second comparison result, and determine the clothing posture corresponding to the human body region based on the second comparison result.

[0045] Step 303: In response to the preset clothing posture corresponding to the human body region being the target clothing posture, the human body region is compared with at least one preset clothing part including key nodes of the human body posture to obtain a third comparison result, and based on the third comparison result, the clothing exposure ratio of the clothing area corresponding to the human body region is determined.

[0046] Step 304 : In response to the clothing exposure ratio of the clothing area reaching a second threshold value under the target clothing posture, the clothing area is selected as a representative clothing area.

[0047] In practical applications, the detected human body region is a rectangular frame containing at least one key human body posture node. The key human body posture nodes may be left eye, right eye, left ear, right ear, nose, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left foot, right foot, and so on. Specifically, the detection may be performed through a pre-trained human body region detection model, and reference may be made to related technologies, which will not be described in detail here.

[0048] The clothing parts are the parts of clothing corresponding to the human body, such as tops and bottoms.

[0049] Since clothing is designed based on the human body, clothing posture and clothing parts can be represented by human body posture key nodes. In implementation, a set of human body posture key nodes for determining clothing posture can be pre-set, and at least one clothing part can be pre-set, that is, at least one preset clothing part is obtained, and each preset clothing part can include human body posture key nodes that characterize the clothing part.

[0050] The clothing exposure ratio may be a ratio of the clothing contained in the clothing region to be exposed vertically in the key frame, wherein the vertical direction refers to a direction corresponding to the height direction of a human body.

[0051] Since clothing needs to be worn on the human body, the posture of the human body area can reflect the posture of the clothing contained in the clothing area corresponding to the human body area. In this embodiment, the preset clothing posture corresponding to the human body area is obtained by comparing the human body area corresponding to the clothing area with the above-mentioned preset human body posture key node set, that is, the posture of the clothing contained in the clothing area corresponding to the human body area is obtained, and then, if the preset clothing posture is the required target clothing posture, the human body area is further compared with the preset clothing part, so as to accurately determine the clothing exposure ratio corresponding to the clothing area, and finally, the clothing area whose clothing exposure ratio reaches the second threshold under the target clothing posture is selected as the representative clothing area.

[0052] In one embodiment, the clothing posture corresponding to the human body region is a front posture, a back posture or an unstable posture. The target clothing posture is a front posture or a back posture. Accordingly, the human body region is compared with a preset human body posture key point set to obtain a second comparison result, and based on the second comparison result, the clothing posture corresponding to the human body region is determined, such as Figure 4 As shown, the specific implementation method may include:

[0053] Step 401: Compare the human body region with a preset set of human body posture key nodes to see if there are human body posture key nodes with the same position.

[0054] Step 402, in response to the comparison result satisfying the first preset condition, determining that the clothing posture corresponding to the human body area is a frontal posture, the first preset condition including: the number of human body posture key nodes located on the left side of the limb among the human body posture key nodes with the same position reaches a third threshold, the number of human body posture key nodes located on the right side of the limb reaches a fourth threshold, the number of human body posture key nodes located on the left side of the face reaches a fifth threshold, the number of human body posture key nodes located on the right side of the face reaches a sixth threshold, and the position of the human body posture key nodes on the left side of the human body is closer to the right side of the key frame than the position of the human body posture key nodes on the right side of the human body.

[0055] Step 403, in response to the comparison result satisfying the second preset condition, determining that the clothing posture corresponding to the human body area is a back posture, the second preset condition includes: the number of human body posture key nodes located on the left side of the limb among the human body posture key nodes with the same position reaches the seventh threshold, the number of human body posture key nodes located on the right side of the limb reaches the eighth threshold, the number of human body posture key nodes located on the left side of the face does not exceed the ninth threshold, the number of human body posture key nodes located on the right side of the face does not exceed the tenth threshold, and the position of the human body posture key node on the left side of the human body is closer to the left side of the key frame than the position of the human body posture key node on the right side of the human body.

[0056] Step 404: In response to the comparison result not satisfying the first preset condition and the second preset condition, determining that the clothing posture corresponding to the human body region is an unstable posture.

[0057] In practical applications, the clothing posture can be a front posture, a back posture or an unstable posture. Among them, postures other than the front posture and the back posture are unstable postures. Exemplarily, unstable postures include side postures. Since the front and back of the clothing are not exposed in the side posture, the information contributed is less, and the clothing posture is a posture when the front posture and the back posture are converted, which is not stable. Therefore, it is not convenient to be used as a condition for representing the clothing area. Based on this, the above-mentioned target clothing posture can be a front posture or a back posture.

[0058] From the perspective of the key frame, in the front posture, one side of the face is generally exposed more, while in the back posture, one side of the face is exposed less. The above-mentioned preset human posture key node set is set in combination with the characteristics of the front posture and the back posture, and based on this, it is compared with the human body area to determine the clothing posture corresponding to the human body area. Exemplarily, the preset human posture key node set may include a human posture key node located on the left side of the limbs, a human posture key node located on the right side of the limbs, a human posture key node located on the left side of the face, and a human posture key node located on the right side of the face.

[0059] If the number of human posture key nodes located on the left side of the limb reaches the third threshold, the number of human posture key nodes located on the right side of the limb reaches the fourth threshold, the number of human posture key nodes located on the left side of the face reaches the fifth threshold, and the number of human posture key nodes located on the right side of the face reaches the sixth threshold, the position of the human posture key nodes on the left side of the body is closer to the right side of the key frame than the position of the human posture key nodes on the right side of the body, which means that one side of the face is displayed more in the key frame, which meets the characteristics of the frontal posture. Based on this, the frontal posture can be accurately determined.

[0060] Similarly, the number of human posture key nodes located on the left side of the limb reaches the seventh threshold, the number of human posture key nodes located on the right side of the limb reaches the eighth threshold, the number of human posture key nodes located on the left side of the face does not exceed the ninth threshold, and the number of human posture key nodes located on the right side of the face does not exceed the tenth threshold. The positions of the human posture key nodes on the left side of the human body are closer to the left side of the key frame than the positions of the human posture key nodes on the right side of the human body, indicating that one side of the face is less displayed in the key frame, which is consistent with the characteristics of the back posture. Based on this, the back posture can be accurately determined.

[0061] If the above two conditions are not met, it is an unstable posture.

[0062] The position of the human posture key node can be represented by the pixel coordinates in the key frame. The pixel coordinates of the human posture key node on the left side of the human body and the pixel coordinates of the human posture key node on the right side of the human body can be used to determine whether the position of the human posture key node on the left side of the human body is closer to the left side of the key frame or closer to the right side of the key frame than the position of the human posture key node on the right side of the human body.

[0063] The xoy coordinate system is established with a vertex of the key frame as the origin and two sides where the vertices intersect as the x-axis and y-axis. For example, the vertex at the upper left corner of the key frame is the origin, and one side of the key frame that is consistent with the height of the human body is the x-axis, and the other side is the y-axis.

[0064] In response to the fact that the sum or average value of the pixel coordinates of the key human posture nodes on the left side of the human body is greater than the sum or average value of the pixel coordinates of the key human posture nodes on the right side of the human body, it is determined that the positions of the key human posture nodes on the left side of the human body are closer to the right side of the key frame than the positions of the key human posture nodes on the right side of the human body.

[0065] In response to the sum or average value of the pixel coordinates of the key human posture nodes on the left side of the human body being less than the sum or average value of the pixel coordinates of the key human posture nodes on the right side of the human body, it is determined that the positions of the key human posture nodes on the left side of the human body are closer to the left side of the key frame than the positions of the key human posture nodes on the right side of the human body.

[0066] Among them, the specific values ​​of the third to tenth thresholds can be set according to actual conditions and are not limited here.

[0067] In this embodiment, based on the comparison result of the human body region with the preset human body posture key node set regarding the human body posture key nodes at the same position, the clothing posture corresponding to the human body region can be accurately determined.

[0068] Generally, the back of clothing is simpler in design and has less information than the front. In order to improve the accuracy of clothing recognition, the second threshold value in the back posture can be set to be smaller than the second threshold value in the front posture, so that in the back posture, the clothing area with a higher clothing exposure ratio is selected as the representative clothing area. In this way, the clothing exposure is more complete and the information is richer. Correspondingly, the accuracy of subsequent recognition is also higher. Exemplarily, the clothing exposure ratio in the front posture is greater than or equal to 50%, and the clothing exposure ratio in the back posture is greater than or equal to 70%.

[0069] In one embodiment, the human body region is compared with at least one preset clothing part including a key node of a human body posture to obtain a third comparison result, and based on the third comparison result, the clothing exposure ratio of the clothing region corresponding to the human body region is determined, such as Figure 5 As shown, the specific implementation method may include:

[0070] Step 501: compare the human body posture key nodes contained in the human body area with the human body posture key nodes contained in the preset clothing part to see if there are human body posture key nodes with the same position, and count the number of human body posture key nodes with the same position to obtain a first number.

[0071] Step 502: Determine the preset clothing part with the largest first quantity as the target clothing part.

[0072] Step 503: Obtain a second corresponding relationship corresponding to the target clothing part, where the second corresponding relationship includes clothing exposure ratios corresponding to each human body posture key node at the same position.

[0073] Step 504: query the clothing exposure ratio corresponding to each human posture key node at the same position from the second corresponding relationship to obtain the clothing exposure ratio of the clothing area corresponding to the human body area.

[0074] In practical applications, the second corresponding relationship corresponding to each clothing part can be preset. Taking the clothing part as a top, for example, the second corresponding relationship corresponding to the top can be set. When the key nodes of the human body posture at the same position include the left shoulder and the right shoulder, it can be considered that the part above the chest is exposed, and the clothing exposure ratio can be set to 30%.

[0075] In this embodiment, the preset clothing part with the largest number of human body posture key nodes at the same position is used as the target clothing part corresponding to the human body area. In this way, it can be accurately determined whether the clothing part worn by the human body area is a top or a bottom, etc. Afterwards, based on the above-mentioned second corresponding relationship corresponding to the target clothing part, the clothing exposure ratio of the clothing area corresponding to the human body area can be accurately obtained.

[0076] In one embodiment, a human body region is compared with at least one preset clothing posture containing human body posture key nodes to obtain a second comparison result, and based on the second comparison result, the preset clothing posture corresponding to the human body region is determined. The specific implementation method may include: comparing the human body posture key nodes contained in the human body region with the human body posture key nodes contained in the preset clothing posture to see whether there are human body posture key nodes in the same position, and counting the second number of human body posture key nodes in the same position; determining the preset clothing posture with the largest second number as the preset clothing posture corresponding to the human body region.

[0077] In this embodiment, the preset clothing posture with the largest number of human posture key nodes at the same position is used as the preset clothing posture corresponding to the human body area. The preset clothing posture corresponding to the human body can accurately reflect the posture of the clothing contained in the clothing area.

[0078] In one embodiment, the clothing area in the same key frame is compared with each human body area to obtain a first comparison result, and a first corresponding relationship is determined based on the first comparison result. The specific implementation method may include: calculating the intersection-and-union ratio of the clothing area in the same key frame and each human body area; removing clothing areas in the key frame whose maximum intersection-and-union ratio is less than a first threshold, or clothing areas in which two or more intersection-and-union ratios are greater than or equal to the first threshold; and selecting the clothing area with the largest intersection-and-union ratio from the clothing areas that have not been removed in the key frame as the clothing area corresponding to the human body area in the first corresponding relationship.

[0079] In actual applications, the detected clothing area and human body area are framed by rectangular frames.

[0080] The intersection-and-union ratio is the ratio of the intersection and the union of two frames. The intersection-and-union ratio can measure the overlap ratio of two frames. Here, the intersection-and-union ratio of the clothing area and the human body area can reflect the overlap ratio of the clothing area and the human body area, that is, the relative size of the overlap. If the maximum values ​​of the above-mentioned intersection-and-union ratios are all less than the first threshold, it means that the intersection-and-union ratios are relatively small. Then, the clothing in the detected clothing area may not be worn on the human body in any human body area. Such clothing areas can be removed without identification. If more than two of the above-mentioned intersection-and-union ratios are greater than or equal to the first threshold, it means that the clothing contained in the detected clothing area may be worn on the human bodies contained in multiple human body areas. However, in fact, a piece of clothing can only be worn on one human body, which is inconsistent with the actual situation. It means that the detected clothing area is problematic and is likely to be an overlapping area caused by occlusion between multiple human bodies, which can be removed. In this way, by removing some clothing areas that interfere with clothing recognition, the accuracy of clothing recognition is further improved.

[0081] In one embodiment, the representative clothing area also satisfies the following conditions:

[0082] The clothing category identified in the representative clothing region is the clothing category expected to be output, and / or the confidence level of the clothing category identified in the representative clothing region exceeds an eleventh threshold.

[0083] The clothing categories can include dresses, pants, coats, underwear, etc.

[0084] In practical applications, the clothing category of the representative clothing area can be identified by using a pre-trained clothing category model. The specific training method can refer to related technologies and is not limited here.

[0085] The confidence of the identified clothing category is the probability that the clothing area is the identified clothing category. The higher the probability, the higher the confidence, that is, the more accurate the possibility that the clothing contained in the clothing area is the identified clothing category, and the clothing category identified as clothing with high confidence is distinguished and selected as the representative clothing area, so that the clothing identification is more accurate. The specific value of the eleventh threshold can be set according to the actual situation and is not limited here.

[0086] In addition, the clothing categories that are expected to be output can be pre-set, based on which, representative clothing areas are selected to better meet the recognition requirements. In this way, the recognition of clothing areas of clothing categories that are not expected to be output, such as pants with low differentiation, can be reduced, thereby improving the efficiency of clothing recognition.

[0087] In one embodiment, the representative clothing area also satisfies the following conditions:

[0088] The clothing attribute identified in the representative clothing region is a clothing attribute expected to be output, and / or the confidence level of the clothing attribute identified in the representative clothing region exceeds a twelfth threshold.

[0089] The clothing attributes may include texture attributes such as polka dots, stripes, and solid colors. In practical applications, the clothing attributes of the representative clothing area can be identified through a pre-trained clothing attribute model. The specific training method can refer to the relevant technology and is not limited here. The confidence of the identified clothing attribute is the probability that the clothing area is the identified clothing attribute. The higher the probability, the higher the confidence, that is, the more accurate the possibility that the clothing contained in the clothing area is the identified clothing attribute, and the clothing with high confidence in the identified clothing attributes is distinguished and selected as the representative clothing area, so that the clothing identification is more accurate.

[0090] The specific value of the twelfth threshold can be set according to actual conditions and is not limited here.

[0091] In addition, the clothing attributes that are expected to be output can be pre-set, and based on this, representative clothing areas are selected to better meet the recognition requirements. In this way, the recognition of clothing areas with clothing attributes that are not expected to be output, such as solid-color pants with low differentiation, can be reduced, thereby improving the efficiency of clothing recognition.

[0092] In one embodiment, before performing clothing recognition based on the selected representative clothing area, the above method may further include: clustering the clothing areas from each key frame based on the features of each clothing area from each key frame to obtain at least one type of clothing area. Accordingly, clothing recognition based on the selected representative clothing area may be specifically implemented by: performing clothing recognition for each type of clothing area based on each representative clothing area included in the type of clothing area to obtain a recognition result for the type of clothing area.

[0093] Clustering is the process of dividing a collection of physical or abstract objects into multiple classes consisting of similar objects. Then, based on the characteristics of each clothing area, each clothing area is clustered, and clothing areas with similar features can be classified into one class. In the time domain, since multiple key frames come from multiple different moments, the clothing areas corresponding to multiple key frames also come from multiple different moments. It can be understood that a class of clothing areas also comes from different moments, and most of the moments at which a class of clothing areas with similar features are located are similar, which means that the time domain consistency is high.

[0094] After clustering, one type of clothing area or multiple types of clothing areas can be obtained. For example, if there is actually only one type of clothing in the video to be identified, then the clothing areas of the key frames at different times all correspond to the same type of clothing. After clustering, one type of clothing area corresponding to the clothing may be obtained, or multiple types of clothing areas corresponding to the clothing may be obtained. If there are multiple types of clothing in the video to be identified, then after clustering, multiple types of clothing areas will be obtained. For example, for the same type of clothing, one type of clothing area corresponding to the clothing may be obtained, or multiple types of clothing areas corresponding to the clothing may be obtained. Assuming that there are two types of clothing in the video to be identified, after clustering, three types of clothing areas are obtained, two of which are clothing areas for one type of clothing, and one is clothing area for another type of clothing.

[0095] Based on this, in this embodiment, the clothing areas corresponding to multiple key frames at different moments in the video to be identified are clustered based on the features of each clothing area, so that clothing areas with similar features and high temporal consistency can be classified into one category. Then, clothing recognition is performed based on representative clothing areas in one category of clothing areas. Compared with clothing recognition based on image features of clothing areas at a single moment, the recognition result is more accurate. In this way, the influence of the clothing area at a single moment is reduced, the temporal consistency of the clothing recognition results in the video is improved, and the cross-time ambiguity problems such as the same clothing at different moments being identified as different clothing, or different clothing at different moments being identified as the same clothing in the related technology are reduced, thereby improving the accuracy of clothing recognition.

[0096] In one embodiment, for each type of clothing area, clothing recognition is performed based on each representative clothing area included in the type of clothing area to obtain an identification result of the type of clothing area. The specific implementation method may include: for each type of clothing area, fusing the image features of each representative clothing area included in the type of clothing area into a target image feature, and performing clothing recognition based on the target image feature to obtain an identification result of the type of clothing area.

[0097] The image features of the clothing area refer to the features in the spatial domain of the key frame, such as texture features, color features, etc. The spatial domain here refers to the space where the pixels of the key frame are located.

[0098] In this embodiment, the image features of each representative clothing area contained in the clothing area are fused into a target image feature, and clothing recognition is performed based on the target image feature to obtain the recognition result of the clothing area. In this way, the image features of each representative clothing area with similar features and high temporal consistency are fused, that is, the image features are fused in the time domain, and clothing recognition is performed based on the target image feature that fuses the image features of each representative clothing area, which reduces the influence of the clothing area at a single moment and improves the temporal consistency of the clothing recognition result in the video. Since the image features of each representative clothing area contained in a clothing area are fused to obtain a clothing area, it is only necessary to perform clothing recognition on one clothing area, which greatly improves the efficiency of clothing recognition compared to performing clothing recognition on each clothing area.

[0099] Exemplarily, the image features of each representative clothing area contained in the clothing area are fused into a target image feature, and the specific implementation method may include: weighted averaging the image features of each representative clothing area in the clothing area to obtain the target image feature. In this embodiment, the image features of each representative clothing area are fused by weighted averaging, which is simple to implement and the obtained target image feature is very accurate.

[0100] In addition, the image features of each representative clothing area contained in this type of clothing area can also be fused into a target image feature through other methods. For example, the image features of each representative clothing area in this type of clothing area can be clustered, and weights can be assigned according to the number of categories in which the image features belong. The larger the number, the greater the weight. Then, based on the assigned weights, the image features of this type of representative clothing area are weighted and summed.

[0101] In one embodiment, for each type of clothing area, clothing recognition is performed based on each representative clothing area included in the type of clothing area to obtain an recognition result for the type of clothing area. The specific implementation method may include: for each type of clothing area, clothing recognition is performed on each representative clothing area included in the type of clothing area to obtain an recognition result for each representative clothing area, the recognition result of the representative clothing area includes at least one candidate style, and at least one candidate style is selected from the recognition results of each representative clothing area in the type of clothing area as the recognition result for the type of clothing area.

[0102] In the present embodiment, clothing recognition is performed based on representative clothing areas included in a category of clothing areas, and at least one candidate style is selected from the recognition results of each representative clothing area in the category of clothing areas as the recognition result of the category of clothing areas. In this way, the factor of temporal consistency is taken into consideration, and clothing recognition is performed based on a category of clothing areas with similar features and high temporal consistency, thereby reducing the influence of clothing areas at a single moment, improving the temporal consistency of clothing recognition results in the video, and reducing cross-time ambiguity problems in related technologies such as the same clothing at different moments being recognized as different clothing styles, or different clothing styles at different moments being recognized as the same clothing style, thereby improving the accuracy of clothing recognition.

[0103] Exemplarily, at least one candidate style is selected from the recognition results of each clothing area in this type of clothing area, and its specific implementation method may include: forming a candidate style set based on the candidate styles in the recognition results of each clothing area in this type of clothing area; calculating the score of each candidate style in the candidate style set; and selecting at least one candidate style from the candidate style set based on the score of each candidate style.

[0104] For example, a type of clothing area L1 includes clothing area a, clothing area b, and clothing area c. The recognition result of clothing area a includes candidate style h1, candidate style h2, candidate style h3, and candidate style h4. The recognition result of clothing area b includes candidate style h1, candidate style h2, candidate style h3, and candidate style h5. The recognition result of clothing area c includes candidate style h1, candidate style h2, candidate style h4, and candidate style h6. Then, the candidate styles in the recognition result of clothing area a, the candidate styles in the recognition result of clothing area b, and the candidate styles in the recognition result of clothing area c form a candidate style set P1, and the candidate style set P1 includes candidate style h1, candidate style h2, candidate style h3, candidate style h4, candidate style h5, and candidate style h6. For each candidate style in the candidate style set P1, the score of the candidate style is calculated, and based on the score of each candidate style, at least one candidate style is selected from the candidate style set P1. For example, candidate style h1 and candidate style h2 are selected.

[0105] The score of the candidate style can represent the rationality of the candidate style as a clothing recognition result. In this embodiment, by calculating the score of the candidate style, a selection basis is provided for the selection of the candidate style, and a reasonable candidate style can be accurately selected, further improving the accuracy of clothing recognition.

[0106] If the score of the candidate style is higher, the rationality is higher, then when selecting at least one candidate style, at least one candidate style with the highest score can be selected, thereby further improving the accuracy of clothing recognition.

[0107] In one embodiment, a score is calculated for each candidate style in a set of candidate styles, and a specific implementation method thereof may include: for each candidate style in the set of candidate styles, based on the similarity between each feature of the candidate style and the same feature of the corresponding clothing area, and / or, a first hit number, and / or, a second hit number, calculating the score of the candidate style; wherein the first hit number is the total hit number of the candidate style in the recognition results of all clothing areas forming the set of candidate styles; and the second hit number is the total hit number of the candidate style in the recognition results of all clothing areas in the video to be recognized.

[0108] The similarity of features is an important factor in measuring the rationality of candidate styles. The more similar the features are, the more likely they are the same style. Therefore, in this embodiment, the similarity of features is used as the calculation basis for scoring candidate styles. The types of features here can include texture features and color features. Color features are added on the basis of texture features. Although texture features contain a certain amount of color features, they are not obvious. Here, color features and texture features are placed side by side as display features, which can increase the proportion of color features, eliminate errors caused by color, and improve recognition accuracy.

[0109] Among them, the first hit refers to the number of occurrences.

[0110] Continuing with the above-mentioned candidate style set P1 as an example, the candidate style set P1 includes candidate style h1, candidate style h2, candidate style h3, candidate style h4, candidate style h5 and candidate style h6.

[0111] The first hit count is the total hit count of the candidate style in the recognition results of clothing area a, clothing area b, and clothing area c that form the candidate style set P1. Taking candidate style h5 as an example, candidate style h5 hits 0 times in the recognition results of clothing area a, candidate style h1 hits 1 time in the recognition results of clothing area b, and candidate style h1 hits 0 times in the recognition results of clothing area c. At this time, the total hit count is 1, that is, the first hit count of candidate style h5 is 1.

[0112] The first hit count can reflect the hit frequency of the candidate style in the recognition results of a type of clothing area. In a type of clothing area, since the features of each clothing area are similar and the temporal consistency is high, it is more likely to be the same style. If the hit frequency of the candidate style in the recognition results of a type of clothing area is higher, that is, more clothing areas have hit the same candidate style, then the candidate style is more likely to be accurate, and it is more reasonable to use the candidate style as the recognition result. Therefore, the first hit count is an important factor in measuring the rationality of the candidate style and can be used as the calculation basis for the candidate style score.

[0113] Assume that after all clothing areas corresponding to the video to be identified are clustered, two types of clothing areas are obtained. In addition to the above-mentioned clothing area L1, another type of clothing area L2 is also included. The other type of clothing area L2 includes clothing area d, clothing area e, and clothing area f. The recognition result of clothing area d includes candidate style h7, candidate style h8, candidate style h9, and candidate style h5. The recognition result of clothing area e includes candidate style h7, candidate style h8, candidate style h9, and candidate style h11. The recognition result of clothing area f includes candidate style h7, candidate style h8, candidate style h10, and candidate style h12.

[0114] The second hit count is the total hit count of the candidate style in the recognition results of each clothing area in one type of clothing area L1 and the recognition results of each clothing area in another type of clothing area L2. Taking candidate style h5 as an example, candidate style h5 hits 0 times in the recognition results of clothing area a, candidate style h1 hits 1 time in the recognition results of clothing area b, candidate style h1 hits 0 times in the recognition results of clothing area c, candidate style h5 hits 1 time in the recognition results of clothing area d, candidate style h5 hits 0 times in the recognition results of clothing area e, and candidate style h5 hits 0 times in the recognition results of clothing area f. At this time, the total hit count is 2 times, that is, the second hit count of candidate style h5 is 2.

[0115] The second hit count can reflect the hit frequency of the candidate style in the recognition results of each clothing area of ​​the entire video to be identified. From the perspective of the entire video to be identified, the greater the difference between the recognition results of different types of clothing areas, the better, because the features of different types of clothing areas are not similar and the temporal consistency is poor, so they are more likely to be different styles. Therefore, we hope that the candidate style is only hit by one type of clothing area. If the candidate style is hit by the recognition results of at least one type of clothing area, the discrimination of different types of styles is poor, and it is unreasonable to use the candidate style as the recognition result. For example, in an extreme case, if there are actually multiple different styles in the entire video to be identified, and a candidate style appears in the recognition results of each type of clothing area in the entire video to be identified, then there is no discrimination at all and it is not suitable to be output as a recognition result. Therefore, the second hit count is also an important factor in measuring the rationality of a candidate style, and can also be used as the calculation basis for the score of the candidate style.

[0116] When calculating the score of candidate style h5, the score of candidate style h5 is calculated based on the similarity between the texture features of candidate style h5 and the texture features of the corresponding clothing area, the similarity between the color features of candidate style h5 and the color features of the corresponding clothing area, the first hit number, and the second hit number.

[0117] In this embodiment, when calculating the score of the candidate style, in addition to considering the important factor of feature similarity, the hit frequency of the candidate style in the recognition results of a category of clothing areas and the hit frequency in the recognition results of each clothing area in the entire video to be recognized are also considered. The factors considered are very comprehensive, and a more accurate candidate style score can be calculated to select a more reasonable candidate style.

[0118] In one embodiment, based on the similarity between each feature of the candidate style and the same feature of the corresponding clothing area, and / or the first hit number, and / or the second hit number, the score of the candidate style is calculated, and its specific implementation method may include: weighted summing the similarity, the first hit number, and the ratio of the first hit number to the second hit number to obtain the score of the candidate style. In this embodiment, by weighted summing the similarity, the first hit number, and the ratio of the first hit number to the second hit number, the score of the candidate style can be calculated more accurately.

[0119] In addition, because we expect that the higher the score of the candidate style, the higher the rationality of the candidate style as a clothing result, as before, the higher the first hit number, the higher the rationality, and the higher the second hit number, the lower the rationality, so here the ratio of the first hit number to the second hit number is taken as a factor in the weighted summation. Based on this, the higher the first hit number, the higher the score of the candidate style, and the higher the second hit number, the lower the score of the candidate style, so the rationality of the candidate style can be accurately evaluated.

[0120] Alternatively, a preset constant may be directly used to replace the first hit number, and the ratio of the preset constant to the second hit number may be used as an item in the weighted summation.

[0121] In one embodiment, the weight of the similarity is greater than the weight of the first hit number, and / or the weight of the first hit number is greater than the weight of the ratio.

[0122] In practical applications, since clothing recognition needs to be based on features, the weight of the feature is the largest, followed by the first hit number, and the weight of the ratio of the first hit number to the second hit number is the smallest. In this way, the output of unreasonable candidate styles can be minimized.

[0123] It should be noted that the above is only an example of a method for calculating the score of the candidate style, and the score of the candidate style can also be calculated by other methods. For example, the similarity corresponding to at least one feature of the candidate style is weighted and summed. Or, the similarity corresponding to at least one feature of the candidate style and the first hit number are weighted and summed. Or, the similarity corresponding to at least one feature of the candidate style and the second hit number are weighted and summed. Or, the first hit number and the second hit number are weighted and summed, and so on.

[0124] It should be noted that the above is only an example of an implementation method for selecting at least one candidate style, and at least one candidate style may also be selected by other methods, such as directly sorting the similarities or first hit counts corresponding to the above candidate styles, and selecting at least one candidate style based on the sorting result. For example, at least one candidate style with the highest similarity or the largest number of first hit counts is selected.

[0125] In one implementation, based on the features of each clothing region from each key frame, each clothing region from each key frame is clustered to obtain at least one type of clothing region, and the specific implementation methods may include the following two:

[0126] The first clustering method: based on the features of each clothing region from each key frame, each clothing region from each key frame is clustered once to obtain at least one type of clothing region corresponding to one cluster.

[0127] The second clustering method: based on the features of each clothing area from each key frame, clustering each clothing area from each key frame to obtain at least one type of clothing area corresponding to the first clustering; taking at least one type of clothing area corresponding to the first clustering as at least one clustering object, clustering again to obtain at least one type of clothing area corresponding to the second clustering.

[0128] Compared with the first clustering method, the second clustering method performs two clusterings, which is called coarse-grained clustering, while the first clustering method only performs one clustering, which is called fine-grained clustering.

[0129] After the first clustering (i.e., fine-grained clustering), there may be more than two types of clothing areas belonging to the same clothing area, so a second clustering (i.e., coarse-grained clustering) is performed here, that is, hierarchical clustering is performed. In this way, the clothing areas with the most similar features and consistent time domains can be further classified into one category. Not only is the classification more accurate, but the information of the fused target image features is also richer, thereby further improving the accuracy of clothing recognition.

[0130] In the first clustering method, for each type of clothing area, clothing recognition is performed based on each representative clothing area included in the clothing area to obtain the recognition result of the clothing area. The implementation method may include the following two methods to obtain the recognition result of the clothing area: First, for each type of clothing area in at least one type of clothing area corresponding to a cluster, the image features of each representative clothing area included in the clothing area are fused into a target image feature, and clothing recognition is performed based on the target image feature to obtain the recognition result of the clothing area. Second, for each type of clothing area in at least one type of clothing area corresponding to a cluster, clothing recognition is performed for each representative clothing area included in the clothing area to obtain the recognition result of each representative clothing area, and the recognition result of the representative clothing area includes at least one candidate style. At least one candidate style is selected from the recognition results of each representative clothing area in the clothing area to serve as the recognition result of the clothing area.

[0131] In the second clustering method, correspondingly, for each type of clothing area, clothing recognition is performed based on each representative clothing area contained in the type of clothing area to obtain the recognition result of the type of clothing area. The implementation method thereof may include the following two methods for obtaining the recognition result of the type of clothing area:

[0132] First, for each type of clothing area in at least one type of clothing area corresponding to the re-clustering, clothing recognition is performed based on each representative clothing area included in the type of clothing area to obtain a recognition result of the type of clothing area.

[0133] Specifically, for each type of clothing area in at least one type of clothing area corresponding to the re-clustering, the image features of each representative clothing area contained in the type of clothing area can be fused into a target image feature, and clothing recognition can be performed based on the target image feature to obtain the recognition result of the type of clothing area. Alternatively, for each type of clothing area in at least one type of clothing area corresponding to the re-clustering, clothing recognition can be performed on each representative clothing area contained in the type of clothing area to obtain the recognition result of each representative clothing area, the recognition result of the representative clothing area includes at least one candidate style, and at least one candidate style is selected from the recognition results of each representative clothing area in the type of clothing area to serve as the recognition result of the type of clothing area.

[0134] In this way, after the two clusterings are completed, clothing recognition is performed, and the recognition speed is fast.

[0135] Second, for each of the at least one clothing area corresponding to the first clustering, clothing recognition is performed based on each representative clothing area contained in the clothing area to obtain the recognition result of the clothing area; for each of the at least one clothing area corresponding to the second clustering, at least one candidate style is selected from the recognition results of each clustering object in the clothing area as the recognition result of the clothing area. The details are as follows:

[0136] First, for each type of clothing area in at least one type of clothing area corresponding to a cluster, the image features of each representative clothing area contained in the clothing area can be fused into a target image feature, and clothing recognition can be performed based on the target image feature to obtain the recognition result of the clothing area. Alternatively, for each type of clothing area in at least one type of clothing area corresponding to a cluster, clothing recognition can be performed on each representative clothing area contained in the clothing area to obtain the recognition result of each representative clothing area, the recognition result of the representative clothing area includes at least one candidate style, and at least one candidate style is selected from the recognition results of each representative clothing area in the clothing area to serve as the recognition result of the clothing area.

[0137] Then, for each type of clothing area in at least one type of clothing area corresponding to the re-clustering, at least one candidate style is selected from the recognition results of each clustering object in the type of clothing area as the recognition result of the type of clothing area.

[0138] In this way, after one clustering, clothing recognition is performed based on the results of fine-grained clustering. Since the number of categories is large and the categories are divided more finely, there are fewer interfering clothing areas for a type of clothing area, and the recognition results are more accurate. After the second clustering, the recognition results of each clustering object in a type of clothing area are fused, and at least one candidate style is selected as the recognition result of this type of clothing area, and the recognition result is also more accurate.

[0139] Specifically, selecting at least one candidate style from the recognition results of each cluster object in the clothing area of ​​this type may include: forming a candidate style set based on the candidate styles in the recognition results of each cluster object in the clothing area of ​​this type; calculating the score of each candidate style in the candidate style set; and selecting at least one candidate style from the candidate style set based on the score of each candidate style. The specific implementation of calculating the score of each candidate style in the candidate style set and selecting at least one candidate style from the candidate style set based on the score of each candidate style can refer to the above-mentioned related embodiments, which will not be repeated here.

[0140] In one implementation, based on the features of each clothing region from each key frame, each clothing region from each key frame is clustered to obtain at least one type of clothing region corresponding to the clustering. The specific implementation method may include:

[0141] For each clothing area, determining an associated clothing area corresponding to the clothing area, and determining associated information corresponding to the clothing area, wherein the associated clothing area comes from a candidate clothing area set, the candidate clothing areas included in the candidate clothing area set are clothing areas whose time represented by the time domain features is earlier than that of the clothing area, the associated clothing area is a candidate clothing area in the candidate clothing area set whose features are most similar to those of the clothing area, and the associated information includes an identifier combination formed by an identifier of the clothing area and an identifier of the corresponding associated clothing area;

[0142] Classify the logo combinations corresponding to each clothing area to obtain at least one type of logo combination, wherein the logo combinations of the same type can be spliced ​​in sequence from beginning to end to form a logo string, and the logos of the spliced ​​parts of the two spliced ​​logo combinations are the same;

[0143] For each type of logo combination, each clothing area corresponding to the logo combination is regarded as a category.

[0144] In one embodiment, the features representing the clothing regions may include time domain features in addition to image features. Based on the features of each clothing region from each key frame, each clothing region from each key frame is clustered to obtain at least one type of clothing region corresponding to the cluster, such as Figure 6As shown, the specific implementation method may include:

[0145] Step 601: for each clothing area, determine an associated clothing area corresponding to the clothing area, and determine associated information corresponding to the clothing area, wherein the associated clothing area is from a candidate clothing area set, the candidate clothing area included in the candidate clothing area set is a clothing area whose time represented by the time domain feature is earlier than that of the clothing area, the associated clothing area is a candidate clothing area in the candidate clothing area set whose feature is most similar to that of the clothing area, and the associated information includes an identifier combination formed by an identifier of the clothing area and an identifier of the corresponding associated clothing area;

[0146] Step 602: classify the identification combinations corresponding to the clothing areas to obtain at least one type of identification combination, wherein the identification combinations of the same type can be spliced ​​in sequence from beginning to end to form an identification string, and the identifications of the spliced ​​parts of the two spliced ​​identification combinations are the same;

[0147] Step 603: for each type of logo combination, the clothing areas corresponding to the logo combination are regarded as one type.

[0148] Among them, the time domain features of the clothing area represent the moment when the key frame where the clothing area is located appears in the video to be identified, that is, the moment when the clothing area appears in the video to be identified. In this way, when clustering is performed, clustering can be performed based on the image features and time domain features of each clothing area. Clustering based on image features can implicitly classify those with higher time domain consistency into one category. In this embodiment, further, the time domain features are used as a display feature, parallel to the image features, as the basis for clustering, which further improves the consistency of each type of clothing area in the time domain, further avoids the situation in the related art where the same clothing at different times is identified as different clothing or different clothing at different times is identified as the same clothing, and improves the accuracy of clothing recognition.

[0149] It should be noted that if a clothing region has no corresponding candidate clothing region, no processing is performed. For example, the clothing region with the earliest time sequence has no corresponding candidate clothing region and no processing is performed.

[0150] For example, key frame 1, key frame 2 and key frame 3 are extracted from the video to be identified in chronological order, wherein two clothing areas are detected in key frame 1, and the labels of the two clothing areas are 1 and 2 respectively; two clothing areas are detected in key frame 2, and the labels of the two clothing areas are 3 and 4 respectively; two clothing areas are detected in key frame 3, and the labels of the two clothing areas are 5 and 6 respectively.

[0151] Take the clothing area 4 in key frame 2 as an example. The associated clothing area of ​​clothing area 2 comes from the candidate clothing area set formed by the two clothing areas in the previous key frame 1 in time sequence. Assuming that the features of clothing area 2 in key frame 1 are most similar to the features of clothing area 4, the associated clothing area of ​​clothing area 4 is clothing area 2. Establish the association information between clothing area 4 and clothing area 2, and the association information includes the identification combination formed by the identification of clothing area 4 and the identification of clothing area 2, for example: 4, 2. Accordingly, the same processing is performed on clothing area 3, clothing area 5 and clothing area 6. Since clothing area 1 and clothing area 2 are the earliest in time sequence, they can be left unprocessed.

[0152] Assume that we also get the identification combination: 3, 1, the identification combination: 5, 4, and the identification combination: 6, 3. Based on this, we can classify the identification combination: 6, 3 and the identification combination: 3, 1 into one category, because these two identification combinations can be spliced ​​to form the identification string 6, 3, 1, and the spliced ​​part of the two identification combinations has the identification "3". Then, the clothing area 6, clothing area 3 and clothing area 1 corresponding to this type of identification combination are classified into one category; and, we can classify the identification combination: 5, 4 and the identification combination: 4, 2 into one category, because these two identification combinations can be spliced ​​to form the identification string 5, 4, 2, and the spliced ​​part of the two identification combinations has the identification "4". Then, the clothing area 5, clothing area 4 and clothing area 2 corresponding to this type of identification combination are classified into one category.

[0153] For a clothing area, the moment represented by the time domain feature of the associated clothing area is earlier than the moment represented by the time domain feature of the clothing area, that is, the time sequence of the associated clothing area is before the clothing area. In this embodiment, a clothing area is associated with the most similar clothing area with a previous time sequence, and the association information is established through the combination of identifiers, that is, a forward mapping relationship from the clothing area to the associated clothing area is established, and then, the clothing areas with the most similar features and consistent time domains are associated through the identifier string formed by connecting the identifier combination end to end, and the associated clothing areas are classified into one category, thereby classifying the clothing areas with the most similar features and consistent time domains into one category.

[0154] In one embodiment, for each clothing area, an associated clothing area corresponding to the clothing area is determined, and a specific implementation method thereof may include: for each clothing area, respectively determining the feature distance between the feature of the clothing area and the feature of each candidate clothing area in the candidate clothing area set, and selecting a candidate clothing area with the smallest feature distance as the associated clothing area. The feature distance is used to measure the similarity between two objects, and the smaller the feature distance, the more similar they are. Therefore, in this embodiment, by determining the feature distance between the feature of the candidate clothing area and the feature of the clothing area, the most similar candidate clothing area is accurately selected as the associated clothing area.

[0155] In one embodiment, characteristic distances between the features of the clothing region and the features of each candidate clothing region in the set of candidate clothing regions are determined respectively, and a specific implementation method thereof may include: calculating the distance between the image features of the clothing region and the image features of the candidate clothing regions as a first distance; calculating the distance between the time domain features of the clothing region and the time domain features of the candidate clothing regions as a second distance; determining a weight coefficient of the first distance based on the second distance; determining a characteristic distance between the features of the clothing region and the features of the candidate clothing regions based on the first distance and the weight coefficient, wherein the larger the second distance, the larger the weight coefficient of the first distance, and the larger the characteristic distance.

[0156] The first distance, i.e., the image feature distance, is used to measure the similarity of the image features of two clothing regions. The smaller the image feature distance, the more similar they are, and the larger the image feature distance, the less similar they are. The second distance, i.e., the temporal interval distance, is used to measure the temporal consistency of two clothing regions. The smaller the temporal interval distance, the higher the temporal consistency, and the larger the temporal interval distance, the lower the temporal consistency.

[0157] In practical applications, although the image feature distance between the clothing region and the candidate clothing region is very small, that is, the two clothing regions are very similar, the time domain interval distance between them is very large, which is likely to come from two different clothing items. Therefore, the feature distance brought by the combination of the image feature distance and the time domain interval distance is relatively large.

[0158] For example, at one moment the back of a pure white T-shirt is shown, and at another moment the front has a large pattern on the chest, but the back is pure white. These are two different pieces of clothing that appear at two different moments far apart. From the perspective of image features, the backs are both pure white. However, due to the long time domain interval, it is generally considered that the feature distance is large.

[0159] Therefore, in this embodiment, the weight coefficient of the first distance is determined based on the second distance, and the characteristic distance is obtained based on the first distance and the weight coefficient. The larger the second distance is, the larger the weight coefficient acting on the first distance is, so that the first distance and its weight coefficient will result in a larger characteristic distance. In this way, the characteristic distance can be accurately obtained.

[0160] Exemplarily, the feature distance may be calculated according to the following formula:

[0161] D=gamma+D1*scale_ratio(1)

[0162] Wherein, D represents the feature distance, D1 represents the first distance, scale_ratio represents the weight coefficient of the first distance, and gamma is a preset constant.

[0163] In one embodiment, the distance between the time domain feature of the clothing area and the time domain feature of the candidate clothing area is calculated as the second distance, and its specific implementation method may include: calculating the time interval between the moment represented by the time domain feature of the clothing area and the moment represented by the time domain feature of the candidate clothing area; if the time interval is less than or equal to the first preset interval, determining the second distance as the first preset interval; if the time interval is greater than the first preset interval and less than the second preset interval, determining the second distance as the time interval; if the time interval is greater than the first preset interval and greater than or equal to the second preset interval, determining the second distance as the second preset interval; wherein the second preset interval is greater than the first preset interval.

[0164] In practical applications, if the time intervals corresponding to the actual clothing area and the candidate clothing area are relatively small, they can be considered to belong to the range of high temporal consistency within a certain range, and the effects on the above-mentioned feature distance are similar, and no distinction can be made. If the actual time interval is relatively large and exceeds a certain range, they belong to the range of low temporal consistency, and the effects on the above-mentioned feature distance are similar, and no distinction can be made. Based on this, in this embodiment, in order to simplify the processing, two time intervals are preset, namely the first preset interval and the second preset interval, wherein one value is larger and the other value is smaller, and the second preset interval can be set to be larger than the first preset interval. Based on this, the time intervals corresponding to the actual clothing area and the candidate clothing area are compared with the first preset interval and the second preset interval, respectively. If the time interval is less than or equal to the first preset interval, they can be considered to belong to the range of high temporal consistency, and the second distance is uniformly determined to be the first preset interval; if the time interval is greater than the first preset interval and less than the second preset interval, the actual temporal consistency degree can be used to determine the second distance as the time interval; if the time interval is greater than the first preset interval and greater than or equal to the second preset interval, it can be considered to belong to the range of low temporal consistency, and the second distance is uniformly determined to be the second preset interval, thereby simplifying the processing.

[0165] Exemplarily, the second distance may be calculated by the following formula:

[0166] tm_coef=min(max(0,tm1–tm2-min_tm_gap),max_tm_gap)*PI / max_tm_gap(2)

[0167] Among them, tm_coef represents the second distance, tm1 represents the moment represented by the time domain characteristics of the clothing area, tm2 represents the moment represented by the time domain characteristics of the candidate clothing area, min() represents the minimum value function, max() represents the maximum value function, min_tm_gap represents the first preset interval, max_tm_gap represents the second preset interval, and PI is a preset constant.

[0168] The specific values ​​of the first preset interval and the second preset interval can be set according to actual conditions.

[0169] Exemplarily, based on the second distance, determining the weight coefficient of the first distance can be specifically implemented by the following formula:

[0170] scale_ratio=(alpha+cosine(tm_coef))*beta(3)

[0171] Among them, cosine() means to find the value of the cosine function, and alpha and beta are preset constants.

[0172] The above-mentioned related embodiments are merely exemplary ways of calculating the characteristic distance. The characteristic distance may also be calculated by other ways. For example, the characteristic distance may be calculated by weighting the first distance and the second distance.

[0173] In practical applications, when performing fine-grained clustering, the image feature distance and the time domain interval distance can be combined to construct a distance measurement matrix between all clothing areas.

[0174] The distance metric matrix D is an N*N matrix, where N represents the number of clothing areas. D(i,j) is an element in the distance metric matrix D, where i and j are the identifiers of the clothing areas in the N clothing areas, j represents the jth clothing area, and D(i,j) represents the feature distance between the features of the i-th clothing area and the j-th clothing area. The distance metric matrix D is constructed using a label mapping algorithm, and the "i,j" contained in D(i,j) is the forward mapping label from clothing area i to clothing area j.

[0175] Then, assign a value to each element D(i,j) in the distance metric matrix D, which can be:

[0176] Calculate the distance between the image feature i_feat of the i-th clothing area and the image feature j_feat of the j-th clothing area to obtain the image feature distance. Specifically, the similarity simi_val(i,j) can be calculated. The similarity can be calculated by the pre-built similarity function simi(). The calculation formula is as follows:

[0177] simi_val(i,j)=simi(i_feat,j_feat)(4)

[0178] Since the greater the similarity, the smaller the distance, we can take -simi_val(i,j) as the image feature distance.

[0179] Calculate the distance between the time-domain feature i_tm of the i-th clothing region and the image feature j_tm of the j-th clothing region to obtain the time-domain interval distance. The calculation formula is as follows:

[0180] tm_coef = min(max(0, i_tm – j_tm - min_tm_gap), max_tm_gap) * PI / max_tm_gap(5)

[0181] Calculate the weight coefficient of the image feature distance. The calculation formula is shown in formula (3).

[0182] Calculate D(i, j) based on the image feature distance and the weight coefficient. The calculation formula is as follows:

[0183] D(i, j) = gamma - simi_val(i, j) * scale_ratio(6)

[0184] Exemplarily, min_tm_gap = 30 and max_tm_gap = 300 can be set.

[0185] Since it is necessary to determine the j-th clothing region with the smallest feature distance (i.e., the most similar feature) in time sequence before the i-th clothing region as the associated clothing region, to meet this condition, the following settings can be adopted to increase the forward mapping label constraint:

[0186] (1) If i = j, it means that the j-th clothing region and the i-th clothing region are the same clothing region, which does not meet the requirements. At this time, set D(i, j) = val1, where val1 is the first preset feature distance. Exemplarily, it is set to 2.0.

[0187] (2) If i_tm = j_tm, it means that the j-th clothing region and the i-th clothing region are at the same moment, which does not meet the requirements. At this time, set D(i, j) = val2, where val2 is the second preset feature distance. Exemplarily, it is set to 3.0.

[0188] (3) If i_tm < j_tm, it means that in time sequence, the j-th clothing region is later than the i-th clothing region, which does not meet the requirements. At this time, set D(i, j) = val3, where val3 is the third preset feature distance. Exemplarily, it is set to 4.0.

[0189] The above first preset feature distance, second preset feature distance, and third preset feature distance are all greater than the calculated feature distance and will not be selected when choosing the smallest feature distance.

[0190] Based on the above distance measurement matrix D, an element D(i, j) with the smallest feature distance corresponding to the i-th clothing area is selected, and based on the forward mapping label contained in the selected D(i, j), the forward mapping relationship i, j (i.e., association information) corresponding to the i-th clothing area is obtained.

[0191] The obtained forward mapping relationships are clustered to obtain fine-grained clustering results. The forward mapping relationships of the same category can be spliced ​​in sequence from beginning to end to form an identification string, and the identifications of the spliced ​​parts of the two spliced ​​forward mapping relationships are the same.

[0192] The above is only an exemplary way of clustering the clothing areas. It is understandable that clustering can also be performed in other ways. For example, the clothing areas can be clustered by clustering algorithms such as K-means, and so on.

[0193] The following takes a specific application scenario as an example to illustrate a clothing recognition method provided by an embodiment of the present invention in more detail.

[0194] like Figure 7 As shown, the clothing recognition method proposed in this embodiment mainly includes several main steps: video key frame extraction, clothing area detection, human posture detection, clothing image selection, clothing clustering, clothing aggregation, point time domain expansion, and bidirectional tracking of recognition results.

[0195] The first step is to extract video key frames.

[0196] In this step, key frames at K different moments are extracted from the video to be identified to obtain key frames (or frames) 1, ..., key frame K. The key frame extraction method can refer to the relevant technology and will not be described here.

[0197] The second step is clothing area detection.

[0198] In this step, clothing region detection is performed on the key frame to obtain a clothing region set.

[0199] The third step is human posture detection and clothing selection.

[0200] In this step, human body region detection is performed on the key frame, and in response to at least one clothing region and at least one human body region being detected in the key frame, the clothing posture corresponding to the human body region is determined (i.e., human body posture detection), and a representative clothing region is selected (i.e., clothing selection). The specific implementation method can refer to the above related embodiments.

[0201] Step 4: Clothing clustering.

[0202] In this step, based on the features of each clothing region from each key frame, each clothing region from each key frame is clustered to obtain at least one type of clothing region. Specifically, fine-grained clustering can be performed, or coarse-grained clustering can be performed, and specific reference is made to the above related embodiments. In implementation, the features of the clothing region can be extracted through a pre-trained CNN model.

[0203] Step 5: Clothing identification.

[0204] In implementation, the features of the candidate styles can be extracted in advance and saved in a database. When performing clothing recognition, clothing recognition can be performed based on the similarity between the features of the clothing area and the features of the candidate styles.

[0205] In this step, for each type of clothing area, the image features of each representative clothing area contained in the type of clothing area are fused into a target image feature, and clothing recognition is performed based on the target image feature to obtain a recognition result of the type of clothing area; or, for each type of clothing area, clothing recognition is performed on each representative clothing area contained in the type of clothing area to obtain a recognition result of each representative clothing area, the recognition result of the representative clothing area includes at least one candidate style, and at least one candidate style is selected from the recognition results of each representative clothing area in the type of clothing area as the recognition result of the type of clothing area.

[0206] For the coarse-grained clustering, for each type of clothing area, at least one candidate style is further selected from the recognition results of cluster objects obtained by each fine-grained clustering in the clothing area as the recognition result of the clothing area. For details, please refer to the above related embodiments.

[0207] Step 6: Expansion of clothing area locations.

[0208] In this step, the time point of the key frame where the clothing area is located is determined, so as to obtain the recognition result of the key frame where the clothing area is located based on the recognition result of the clothing area. In this way, the recognition result of the clothing area is extended to the time points of all key frames contained in the category where the clothing area is located, thereby improving the clothing recall duration at the video level.

[0209] Step 7: Bidirectional tracking of video sequences.

[0210] In this step, the video frames before the key frame and the video frames after the key frame in the video to be identified can be subjected to region detection, the video frames in the video frame sequence of the video to be identified where the clothing region appears can be determined, the time point where the clothing region appears can be obtained, and the position information of the clothing region in the spatial domain can be determined, so as to obtain the time point and position information of the clothing with the same or similar style appearing in the video to be identified. Based on the determined time point and position information, the display of the clothing identification result is controlled.

[0211] This solution aims to solve the problem of cross-time style ambiguity when using image recognition solutions for video clothing recognition. It selects representative clothing areas to reduce recognition errors that may be caused by unstable postures and incomplete noisy clothing areas, improves the interpretability of clothing recognition results, and aggregates the recognition results based on fine-grained clustering results to further reduce the impact of misrecognition results and improve the accuracy of recognition results. It also expands the time domain points based on the recognition results to increase the recall time.

[0212] Figure 8 FIG. 1 is a schematic diagram of an exemplary clothing recognition device provided by an embodiment of the present invention. Figure 8 As shown, the device 800 comprises:

[0213] A key frame extraction module 801 is used to extract key frames at different moments from the video to be identified;

[0214] The region detection module 802 is used to detect clothing regions and human body regions on the key frames;

[0215] The representative selection module 803 is used to select a representative clothing area from the at least one clothing area in response to detecting at least one clothing area and at least one human body area in the key frame, where the representative clothing area meets the following conditions: the overlap ratio of the representative clothing area and the human body area reaches a first threshold, the representative clothing area is in a target clothing posture, and the clothing exposure ratio reaches a second threshold under the target clothing posture;

[0216] The clothing recognition module 804 is used to perform clothing recognition based on the selected representative clothing area.

[0217] In one embodiment, the representative selection module is specifically configured to:

[0218] Comparing the clothing region in the same key frame with each human body region to obtain a first comparison result, and determining a first corresponding relationship based on the first comparison result, the first corresponding relationship including the human body region and the clothing region whose overlap ratio reaches a first threshold;

[0219] For each human body region in the first corresponding relationship, the human body region is compared with a preset human body posture key node set to obtain a second comparison result, and based on the second comparison result, a clothing posture corresponding to the human body region is determined;

[0220] In response to the clothing posture corresponding to the human body region being the target clothing posture, comparing the human body region with at least one preset clothing part including key nodes of the human body posture to obtain a third comparison result, and determining the clothing exposure ratio of the clothing area corresponding to the human body region based on the third comparison result;

[0221] In response to the clothing exposure ratio of the clothing region reaching a second threshold under the target clothing posture, the clothing region is selected as a representative clothing region.

[0222] In one embodiment, the representative selection module is specifically configured to:

[0223] Compare the human body region with the preset human body posture key node set to see if there are human body posture key nodes with the same position;

[0224] In response to the comparison result satisfying the first preset condition, determining that the clothing posture corresponding to the human body region is a frontal posture, the first preset condition comprising: the number of human body posture key nodes located on the left side of the limbs among the human body posture key nodes with the same position reaches a third threshold, the number of human body posture key nodes located on the right side of the limbs reaches a fourth threshold, the number of human body posture key nodes located on the left side of the face reaches a fifth threshold, the number of human body posture key nodes located on the right side of the face reaches a sixth threshold, and the positions of the human body posture key nodes on the left side of the human body are closer to the right side of the key frame than the positions of the human body posture key nodes on the right side of the human body;

[0225] In response to the comparison result satisfying the second preset condition, determining that the clothing posture corresponding to the human body region is a back posture, the second preset condition includes: the number of human body posture key nodes located on the left side of the limb among the human body posture key nodes with the same position reaches the seventh threshold, the number of human body posture key nodes located on the right side of the limb reaches the eighth threshold, the number of human body posture key nodes located on the left side of the face does not exceed the ninth threshold, the number of human body posture key nodes located on the right side of the face does not exceed the tenth threshold, and the position of the human body posture key node on the left side of the human body is closer to the left side of the key frame than the position of the human body posture key node on the right side of the human body;

[0226] In response to the comparison result not satisfying the first preset condition and the second preset condition, it is determined that the clothing posture corresponding to the human body region is an unstable posture.

[0227] In one embodiment, the representative selection module is specifically configured to:

[0228] Comparing the human body posture key nodes included in the human body region with the human body posture key nodes included in the preset clothing part to determine whether there are human body posture key nodes with the same position, and counting the number of human body posture key nodes with the same position to obtain a first number;

[0229] Determine the preset clothing part with the largest first number as the target clothing part;

[0230] Acquire a second correspondence relationship corresponding to the target clothing part, the second correspondence relationship including clothing exposure ratios corresponding to key human posture nodes at the same position;

[0231] From the second corresponding relationship, the clothing exposure ratio corresponding to each human posture key node at the same position is queried to obtain the clothing exposure ratio of the clothing area corresponding to the human body area.

[0232] In one embodiment, the representative selection module is specifically configured to:

[0233] Calculate the intersection-over-union ratio of the clothing area and each human body area in the same key frame;

[0234] In the key frame, clothing regions whose maximum values ​​of intersection-and-union ratios are less than a first threshold value, or clothing regions whose intersection-and-union ratios have more than two intersection-and-union ratios greater than or equal to the first threshold value, are removed;

[0235] From the clothing regions that are not removed in the key frame, a clothing region with the largest intersection-over-union ratio is selected as the clothing region corresponding to the human body region in the first corresponding relationship.

[0236] In one embodiment, the representative clothing area also satisfies the following conditions:

[0237] The clothing category identified in the clothing region is the clothing category expected to be output, and / or the confidence level of the clothing category identified in the clothing region exceeds an eleventh threshold;

[0238] and / or,

[0239] The clothing attribute identified in the clothing region is the clothing attribute expected to be output, and / or the confidence level of the clothing attribute identified in the clothing region exceeds a twelfth threshold.

[0240] In one embodiment, it further includes:

[0241] A region clustering module, for clustering the clothing regions from each key frame based on the features of the clothing regions from each key frame before performing clothing recognition based on the selected representative clothing regions, so as to obtain at least one type of clothing region;

[0242] Represents the selection module, specifically used for:

[0243] For each type of clothing area, clothing recognition is performed based on each representative clothing area included in the type of clothing area to obtain a recognition result of the type of clothing area.

[0244] In one embodiment, the clothing recognition module is specifically used to:

[0245] For each type of clothing area, the image features of each representative clothing area contained in the type of clothing area are fused into a target image feature, and clothing recognition is performed based on the target image feature to obtain a recognition result of the type of clothing area;

[0246] Alternatively, for each type of clothing area, clothing recognition is performed on each representative clothing area contained in that type of clothing area to obtain an recognition result for each representative clothing area, the recognition result of the representative clothing area includes at least one candidate style, and at least one candidate style is selected from the recognition results of each representative clothing area in that type of clothing area as the recognition result of that type of clothing area.

[0247] In one implementation, the regional clustering module is specifically used to:

[0248] Based on the features of each clothing region from each key frame, clustering each clothing region from each key frame to obtain at least one type of clothing region corresponding to the clustering;

[0249] At least one type of clothing area corresponding to the first clustering is taken as at least one clustering object, and clustering is performed again to obtain at least one type of clothing area corresponding to the second clustering.

[0250] In one embodiment, the clothing recognition module is specifically used to:

[0251] For each type of clothing area in at least one type of clothing area corresponding to the re-clustering, clothing recognition is performed based on each representative clothing area included in the type of clothing area to obtain a recognition result of the type of clothing area;

[0252] Alternatively, for each type of clothing area in at least one type of clothing area corresponding to the first clustering, clothing recognition is performed based on each representative clothing area contained in the type of clothing area to obtain an recognition result of the type of clothing area; for each type of clothing area in at least one type of clothing area corresponding to the second clustering, at least one candidate style is selected from the recognition results of each clustering object in the type of clothing area as the recognition result of the type of clothing area.

[0253] In one implementation, the regional clustering module is specifically used to:

[0254] For each clothing area, determining an associated clothing area corresponding to the clothing area, and determining associated information corresponding to the clothing area, wherein the associated clothing area comes from a candidate clothing area set, the candidate clothing areas included in the candidate clothing area set are clothing areas whose time represented by the time domain features is earlier than that of the clothing area, the associated clothing area is a candidate clothing area in the candidate clothing area set whose features are most similar to those of the clothing area, and the associated information includes an identifier combination formed by an identifier of the clothing area and an identifier of the corresponding associated clothing area;

[0255] Classify the logo combinations corresponding to each clothing area to obtain at least one type of logo combination, wherein the logo combinations of the same type can be spliced ​​in sequence from beginning to end to form a logo string, and the logos of the spliced ​​parts of the two spliced ​​logo combinations are the same;

[0256] For each type of logo combination, each clothing area corresponding to the logo combination is regarded as a category.

[0257] The functions of the modules in the devices provided in the embodiments of the present invention can be found in the corresponding descriptions in the above-mentioned clothing recognition method embodiments, which will not be described in detail here.

[0258] The embodiment of the present invention further provides an electronic device, such as Fig. 9 As shown, it includes a processor 901, a communication interface 902, a memory 903 and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0259] Memory 903, used for storing computer programs;

[0260] The processor 901 is used to execute the program stored in the memory 903 to implement the following steps:

[0261] Extract key frames at multiple different moments from the video to be identified;

[0262] Perform clothing area and human body area detection on key frames;

[0263] In response to detecting at least one clothing region and at least one human body region in the key frame, selecting a representative clothing region from the at least one clothing region, wherein the representative clothing region satisfies the following conditions: an overlap ratio between the representative clothing region and the human body region reaches a first threshold, the representative clothing region is in a target clothing posture, and a clothing exposure ratio reaches a second threshold under the target clothing posture;

[0264] Based on the selected representative clothing area, clothing recognition is performed.

[0265] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0266] The communication interface is used for communication between the above terminal and other devices.

[0267] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0268] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0269] In another embodiment provided by the present invention, a computer-readable storage medium is provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the clothing recognition method described in any one of the above embodiments.

[0270] In another embodiment of the present invention, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to execute any clothing recognition method described in the above embodiments.

[0271] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0272] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0273] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0274] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A clothing recognition method, characterized in that: include: Extract key frames at multiple different moments from the video to be identified; Performing clothing area and human body area detection on the key frame; In response to detecting at least one clothing region and at least one human body region in the key frame, selecting a representative clothing region from the at least one clothing region, wherein the representative clothing region satisfies the following conditions: an overlap ratio between the representative clothing region and the human body region reaches a first threshold, the representative clothing region is in a target clothing posture, and a clothing exposure ratio reaches a second threshold under the target clothing posture; Clothing recognition is performed based on the selected representative clothing area.

2. The method according to claim 1, characterized in that The human body region includes a key human body posture node at at least one position, and the selecting a representative clothing region from the at least one clothing region includes: Comparing the clothing region with each of the human body regions in the same key frame to obtain a first comparison result, and determining a first corresponding relationship based on the first comparison result, wherein the first corresponding relationship includes the human body region and the clothing region whose overlap ratio reaches a first threshold; For each of the human body regions in the first corresponding relationship, the human body region is compared with a preset set of key human body posture nodes to obtain a second comparison result, and based on the second comparison result, a clothing posture corresponding to the human body region is determined; In response to the clothing posture corresponding to the human body region being a target clothing posture, comparing the human body region with at least one preset clothing part including a key node of the human body posture to obtain a third comparison result, and determining the clothing exposure ratio of the clothing region corresponding to the human body region based on the third comparison result; In response to the clothing exposure ratio of the clothing area reaching a second threshold value under the target clothing posture, the clothing area is selected as the representative clothing area.

3. The method according to claim 2, characterized in that The clothing posture corresponding to the human body region is a front posture, a back posture or an unstable posture, the target clothing posture is the front posture or the back posture, and the human body region is compared with a preset human body posture key point set to obtain a second comparison result, and based on the second comparison result, the clothing posture corresponding to the human body region is determined, including: Comparing the human body region with the preset human body posture key node set to see whether there are human body posture key nodes with the same position; In response to the comparison result satisfying the first preset condition, determining that the clothing posture corresponding to the human body region is a frontal posture, the first preset condition comprising: the number of the human body posture key nodes located on the left side of the limbs among the human body posture key nodes with the same position reaches a third threshold, the number of the human body posture key nodes located on the right side of the limbs reaches a fourth threshold, the number of the human body posture key nodes located on the left side of the face reaches a fifth threshold, the number of the human body posture key nodes located on the right side of the face reaches a sixth threshold, and the positions of the human body posture key nodes on the left side of the human body are closer to the right side of the key frame than the positions of the human body posture key nodes on the right side of the human body; In response to the comparison result satisfying the second preset condition, determining that the clothing posture corresponding to the human body region is a back posture, the second preset condition includes: the number of the human body posture key nodes located on the left side of the limbs among the human body posture key nodes with the same position reaches a seventh threshold, the number of the human body posture key nodes located on the right side of the limbs reaches an eighth threshold, the number of the human body posture key nodes located on the left side of the face does not exceed a ninth threshold, the number of the human body posture key nodes located on the right side of the face does not exceed a tenth threshold, and the positions of the human body posture key nodes on the left side of the human body are closer to the left side of the key frame than the positions of the human body posture key nodes on the right side of the human body; In response to the comparison result not satisfying the first preset condition and the second preset condition, it is determined that the clothing posture corresponding to the human body region is an unstable posture.

4. The method according to claim 2, characterized in that: The step of comparing the human body region with at least one preset clothing part including a key node of a human body posture to obtain a third comparison result, and determining the clothing exposure ratio of the clothing region corresponding to the human body region based on the third comparison result, comprises: Comparing the human body posture key nodes included in the human body region with the human body posture key nodes included in the preset clothing part to determine whether there are human body posture key nodes with the same position, and counting the number of human body posture key nodes with the same position to obtain a first number; determining the preset clothing parts with the largest first number as target clothing parts; Acquire a second correspondence relationship corresponding to the target clothing part, wherein the second correspondence relationship includes clothing exposure ratios corresponding to key human posture nodes at the same position; The clothing exposure ratio corresponding to each of the human body posture key nodes at the same position is queried from the second corresponding relationship to obtain the clothing exposure ratio of the clothing area corresponding to the human body area.

5. The method according to claim 2, characterized in that: The step of comparing the clothing region in the same key frame with each of the human body regions to obtain a first comparison result, and determining a first corresponding relationship based on the first comparison result, includes: Calculating the intersection-over-union ratio of the clothing region and each of the human body regions in the same key frame; Removing the clothing regions in the key frames whose maximum values ​​of the intersection-and-union ratios are less than the first threshold, or the clothing regions whose two or more intersection-and-union ratios are greater than or equal to the first threshold; The clothing region having the largest intersection-over-union ratio is selected from the clothing regions that are not removed in the key frame as the clothing region corresponding to the human body region in the first corresponding relationship.

6. The method according to claim 1, characterized in that The representative clothing area also meets the following conditions: The clothing category identified in the representative clothing area is the clothing category expected to be output, and / or the confidence of the clothing category identified in the representative clothing area exceeds an eleventh threshold; and / or, The clothing attributes identified in the representative clothing area are clothing attributes expected to be output, and / or the confidence level of the clothing attributes identified in the representative clothing area exceeds a twelfth threshold.

7. The method according to claim 1, characterized in that Before performing clothing recognition based on the selected representative clothing area, the method further includes: Based on the features of each clothing region from each key frame, clustering each clothing region from each key frame to obtain at least one type of clothing region; The performing clothing recognition based on the selected representative clothing area includes: For each type of clothing area, clothing recognition is performed based on each of the representative clothing areas included in the type of clothing area to obtain a recognition result of the type of clothing area.

8. The method according to claim 7, characterized in that For each type of clothing area, clothing recognition is performed based on each representative clothing area included in the type of clothing area to obtain a recognition result of the type of clothing area, including: For each type of clothing area, the image features of each representative clothing area contained in the type of clothing area are fused into a target image feature, and clothing recognition is performed based on the target image feature to obtain a recognition result of the type of clothing area; Alternatively, for each type of clothing area, clothing recognition is performed on each of the representative clothing areas contained in that type of clothing area to obtain an recognition result for each of the representative clothing areas, the recognition results of the representative clothing areas including at least one candidate style, and at least one candidate style is selected from the recognition results of each of the representative clothing areas in that type of clothing area as the recognition result of that type of clothing area.

9. The method according to claim 7, characterized in that: The clustering of the clothing regions from the key frames based on the features of the clothing regions from the key frames to obtain at least one type of clothing regions includes: Based on the features of each clothing region from each key frame, clustering each clothing region from each key frame to obtain at least one type of clothing region corresponding to the clustering; The at least one type of clothing area corresponding to the first clustering is taken as at least one clustering object, and clustering is performed again to obtain the at least one type of clothing area corresponding to the second clustering.

10. The method according to claim 9, characterized in that For each type of clothing area, clothing recognition is performed based on each representative clothing area included in the type of clothing area to obtain a recognition result of the type of clothing area, including: For each type of clothing area in the at least one type of clothing area corresponding to the re-clustering, clothing recognition is performed based on each of the representative clothing areas contained in the type of clothing area to obtain a recognition result of the type of clothing area; Alternatively, for each type of clothing area in at least one type of clothing area corresponding to the first clustering, clothing recognition is performed based on each of the representative clothing areas contained in that type of clothing area to obtain an recognition result of that type of clothing area; for each type of clothing area in at least one type of clothing area corresponding to the second clustering, at least one candidate style is selected from the recognition results of each of the clustering objects in that type of clothing area as the recognition result of that type of clothing area.

11. The method according to claim 9, characterized in that The step of clustering the clothing regions from the key frames based on the features of the clothing regions from the key frames to obtain at least one type of clothing region corresponding to the clustering comprises: For each clothing area, determining an associated clothing area corresponding to the clothing area, and determining associated information corresponding to the clothing area, wherein the associated clothing area is from a set of candidate clothing areas, the candidate clothing areas included in the set of candidate clothing areas are clothing areas whose time represented by the time domain features is earlier than that of the clothing area, the associated clothing area is a candidate clothing area in the set of candidate clothing areas whose features are most similar to those of the clothing area, and the associated information includes an identifier combination formed by an identifier of the clothing area and an identifier of the corresponding associated clothing area; Classifying the identification combinations corresponding to the clothing areas to obtain at least one type of identification combination, wherein the identification combinations of the same type can be spliced ​​in sequence from beginning to end to form an identification string, and the identifications of the spliced ​​parts of the two spliced ​​identification combinations are the same; For each type of logo combination, the clothing areas corresponding to the logo combination are regarded as one type.

12. A clothing recognition device, characterized in that: include: A key frame extraction module is used to extract key frames at multiple different moments from the video to be identified; A region detection module, used for detecting clothing regions and human body regions on the key frames; a representative selection module, configured to select a representative clothing region from the at least one clothing region in response to detecting at least one clothing region and at least one human body region in the key frame, wherein the representative clothing region satisfies the following conditions: an overlap ratio between the representative clothing region and the human body region reaches a first threshold, the representative clothing region is in a target clothing posture, and a clothing exposure ratio reaches a second threshold under the target clothing posture; The clothing recognition module is used to perform clothing recognition based on the selected representative clothing area.

13. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1-11 when executing a program stored in a memory.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images