Clothing matching method and device, electronic equipment and storage medium

Abstract

The clothing matching method, apparatus, electronic device, and storage medium provided in this invention include: performing object detection and clothing detection bounding boxes on a first image; marking object detection bounding boxes for objects and clothing detection bounding boxes for clothing on the first image; taking the object corresponding to each object detection bounding box as a first object; calculating the matching value between the target object detection bounding box corresponding to the first object and each clothing detection bounding box; and determining the clothing that matches the first object based on multiple matching values. The matching value calculated in this invention represents the proportion of the intersection region of the target object detection bounding box and the clothing detection bounding box within the clothing detection bounding box, while the traditional IOU calculation method represents the proportion of the intersection region within the union region. Compared to the IOU calculation method, this effectively reduces the impact of human body integrity on the occlusion score (i.e., the matching value) during the matching process, improves the influence of the clothing detection bounding box (i.e., the clothing bounding box) during the matching process, and thus improves the accuracy of clothing-human body matching.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, apparatus, electronic device and storage medium for matching clothing. Background Technology

[0002] In video scenes, by analyzing the type and style of clothing worn by characters, we can gain a deeper understanding of their image and personality, which can help recommend suitable products to them or help analyze their positioning in the whole drama and the direction of the story. As a prerequisite for analyzing characters through clothing, it is necessary to match the characters with the corresponding clothing.

[0003] Currently, in order to obtain the matching relationship between clothing and people, AI (Artificial Intelligence) models can be used to detect clothing boxes and human body boxes in the video. Based on this, the IOU (Intersection over Union) between all clothing boxes and human body boxes is calculated. When the matching IOU is greater than a preset threshold, the corresponding clothing-human body match is considered successful. The IOU calculation formula is: IOU = Intersection area of ​​clothing box and human body box / Union area of ​​clothing box and human body box.

[0004] As can be seen from the above IOU calculation formula, when the detected clothing frame is incomplete, the area occupied by the clothing frame is smaller, while the area occupied by the human body frame is larger. This results in the IOU value calculated for incomplete clothing being much smaller than the IOU value calculated for complete clothing. In addition to the completeness of the clothing frame affecting the IOU value, the completeness of the human body frame also affects the IOU value. With a fixed clothing frame, the IOU value calculated for a human body frame that shows the whole body is smaller than the IOU value calculated for a human body frame that shows only half the body. Due to the above reasons, the calculated IOU may be smaller than the preset threshold, thus filtering out clothing that matches the person, resulting in a large number of incorrect matching results and reducing the accuracy of clothing-human body matching. Summary of the Invention

[0005] Therefore, embodiments of the present invention provide a clothing matching method, apparatus, electronic device, and storage medium, which can improve the accuracy of clothing-human body matching.

[0006] In a first aspect, embodiments of the present invention provide a clothing matching method, wherein the method includes:

[0007] Perform object detection on the first image and mark the object detection boxes on the first image;

[0008] Clothing detection is performed on the first image, and clothing detection boxes are marked on the first image; wherein, the first image is a single frame image in the video sequence that includes objects and clothing, and clothing includes upper clothing and / or lower clothing.

[0009] For each object detection box, the object corresponding to that object is designated as the first object. For each first object, the following operations are performed:

[0010] Calculate the matching value between the target object detection box corresponding to the first object and each clothing detection box; whereby the matching value is used to characterize the proportion of the intersection area between the target object detection box and the clothing detection box in the clothing detection box;

[0011] The clothing that matches the first object is determined based on multiple matching values.

[0012] In one possible implementation, object detection is performed on the first image, and object detection boxes are marked on the first image, including:

[0013] The first image is input into the first neural network model to extract object location features from the first image using the first neural network model, thereby obtaining the first feature map of the first image;

[0014] For each point on the first feature map, generate multiple first anchor boxes, where each first anchor box has a different size;

[0015] For each first anchor box, calculate the first confidence that an object exists in the first anchor box;

[0016] The first target anchor box with a first confidence level greater than or equal to the first threshold is retained, and the region enclosed by the first target anchor box is determined as the object detection box of the object.

[0017] In one possible implementation, clothing detection is performed on the first image, and clothing detection boxes are marked on the first image, including:

[0018] The first image is input into the second neural network model to extract clothing location features from the first image using the second neural network model, thereby obtaining the second feature map of the first image;

[0019] For each point on the second feature map, generate multiple second anchor boxes, where each second anchor box has a different size;

[0020] For each second anchor box, calculate the second confidence level that clothing exists in the second anchor box;

[0021] The second target anchor box with a second confidence level greater than or equal to the first threshold is retained, and the area enclosed by the second target anchor box is determined as the clothing detection box.

[0022] In one possible implementation, calculating the matching value between the target object detection box corresponding to the first object and each clothing detection box includes:

[0023] In the image coordinate system of the first image, obtain the coordinate information of the first detection box of the target object detection box and the coordinate information of the second detection box of each clothing detection box;

[0024] For each clothing detection frame, a matching value is calculated based on the coordinate information of the first detection frame and the coordinate information of the second detection frame.

[0025] In one possible implementation, the first detection box coordinate information includes the first coordinate information of the upper left corner position and the second coordinate information of the lower right corner position of the target object detection box, and the second detection box coordinate information includes the third coordinate information of the upper left corner position and the fourth coordinate information of the lower right corner position of the clothing detection box.

[0026] The matching value is calculated based on the coordinate information of the first and second detection boxes, including:

[0027] The first area of ​​the clothing detection frame is determined based on the third and fourth coordinate information;

[0028] Based on the first, second, third, and fourth coordinate information, determine the fifth coordinate information of the upper left corner and the sixth coordinate information of the lower right corner of the intersection area of ​​the target object detection box and the clothing detection box;

[0029] The second area of ​​the intersection region is determined based on the fifth and sixth coordinate information;

[0030] The matching value is calculated based on the first area and the second area.

[0031] In one possible implementation, determining the fifth coordinate information of the upper-left corner and the sixth coordinate information of the lower-right corner of the intersection region based on the first coordinate information, the second coordinate information, the third coordinate information, and the fourth coordinate information includes:

[0032] The fifth coordinate information is determined based on the first and third coordinate information;

[0033] The sixth coordinate information is determined based on the second and fourth coordinate information.

[0034] In one possible implementation, determining the fifth coordinate information based on the first coordinate information and the third coordinate information includes:

[0035] The maximum horizontal axis coordinate is selected from the first horizontal axis coordinate of the first coordinate information and the third horizontal axis coordinate of the third coordinate information and determined as the fifth horizontal axis coordinate of the fifth coordinate information;

[0036] The largest ordinate is selected from the first ordinate of the first coordinate information and the third ordinate of the third coordinate information and determined as the fifth ordinate of the fifth coordinate information.

[0037] In one possible implementation, determining the sixth coordinate information based on the second and fourth coordinate information includes:

[0038] The minimum horizontal axis coordinate is selected from the second horizontal axis coordinate of the second coordinate information and the fourth horizontal axis coordinate of the fourth coordinate information and determined as the sixth horizontal axis coordinate of the sixth coordinate information;

[0039] The smallest ordinate is selected from the second ordinate of the second coordinate information and the fourth ordinate of the fourth coordinate information and determined as the sixth ordinate of the sixth coordinate information.

[0040] In one possible implementation, determining the clothing that matches the first object based on multiple matching values ​​includes:

[0041] Compare each matching value with the preset matching threshold;

[0042] The clothing in the clothing detection box corresponding to the matching value greater than the preset matching threshold is identified as the clothing that matches the first object.

[0043] Secondly, embodiments of the present invention provide a keyframe extraction method, wherein the method includes:

[0044] Obtain the target video sequence;

[0045] Extract single-frame images from the target video sequence;

[0046] Perform clothing matching on a single frame image to determine the matching clothing of the target object in the single frame image;

[0047] The single-frame image that matches the clothing and meets the target conditions is determined as the keyframe;

[0048] The clothing matching method described above is included in the step of matching clothing to a single frame image.

[0049] Thirdly, embodiments of the present invention provide a clothing matching device, wherein the device includes:

[0050] The first detection module is used to perform object detection on the first image and mark the object detection boxes of 5 objects on the first image;

[0051] The second detection module is used to detect clothing in the first image and mark the clothing detection box on the first image; wherein, the first image is a single frame image in the video sequence that includes objects and clothing, and the clothing includes upper clothing and / or lower clothing.

[0052] The execution module is used to treat the object corresponding to each object detection box as the first object, and perform the following operations for each first object:

[0053] The calculation module is used to calculate the matching value between the target object detection box corresponding to the first object and each clothing detection box; wherein, the matching value is used to represent the proportion of the intersection area between the target object detection box and the clothing detection box in the clothing detection box;

[0054] The determination module is used to determine the clothing that matches the first object based on multiple matching values.

[0055] 5. In a fourth aspect, embodiments of the present invention provide an electronic device, comprising: a processor and

[0056] The memory and processor are used to execute the clothing matching and keyframe extraction programs stored in the memory to implement the steps of the method described above.

[0057] Fifthly, embodiments of the present invention provide a storage medium, wherein the storage medium stores...

[0058] One or more programs, which can be executed by one or more processors, implement the steps of the above method in 0.

[0059] The clothing matching method, apparatus, electronic device, and storage medium provided in this invention include: performing object detection on a first image and marking object detection boxes on the first image; performing clothing detection on the first image and marking clothing detection boxes on the first image; and storing each...

[0060] Each object detection box corresponds to an object as a first object. For each first object, the following five operations are performed: calculate the matching value between the target object detection box corresponding to the first object and each clothing detection box;

[0061] The clothing that matches the first object is determined based on multiple matching values. In this invention, the matching value calculated based on the target object detection box and the clothing detection box is used to characterize the proportion of the intersection area of ​​the target object detection box and the clothing detection box within the clothing detection box. In contrast, the traditional IOU calculation method is used to characterize the proportion of the intersection area within the union area. Compared to the IOU calculation method, since the denominator is modified to use the area of ​​the clothing detection box for calculation, the intersection score of the clothing detection box and the human body detection box will not change with the change of human body integrity. This effectively reduces the influence of human body integrity on the occlusion score, i.e., the matching value, during the matching process, and improves the influence of the clothing detection box, i.e., the clothing box, during the matching process, thereby improving the accuracy of clothing-human body matching. Attached Figure Description

[0062] Figure 1This is a schematic diagram of the hardware environment for a clothing matching method provided in an embodiment of the present invention;

[0063] Figure 2 This is a flowchart illustrating a clothing matching method provided in an embodiment of the present invention;

[0064] Figure 3 A schematic diagram of an image coordinate system for a first image provided in an embodiment of the present invention;

[0065] Figure 4 A flowchart illustrating a keyframe extraction method provided in an embodiment of the present invention;

[0066] Figure 5 A block diagram illustrating an embodiment of a clothing matching device provided by the present invention;

[0067] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] To facilitate understanding of the embodiments of the present invention, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0070] In this embodiment, the above-described clothing matching method can be applied to, for example... Figure 1 The hardware environment shown consists of terminal 101 and server 103. Figure 1 As shown, server 103 is connected to terminal 101 via a network and can be used to provide services (such as clothing matching services) to the terminal or clients installed on the terminal. Database 105 can be set up on the server or independently of the server to provide data storage services for server 103. The network mentioned above includes, but is not limited to, wide area network, metropolitan area network or local area network. Terminal 101 includes, but is not limited to, PC, mobile phone, tablet computer, etc.

[0071] The clothing matching method in this embodiment can be executed by server 103, or it can be executed jointly by server 103 and terminal 101, such as... Figure 2 As shown, the method may include the following steps:

[0072] Step 201: Perform object detection on the first image and mark the object detection bounding boxes on the first image;

[0073] The first image is a single-frame image of an object in a video sequence; specifically, the first image is a single-frame image in a video sequence, which includes, but is not limited to, videos in various existing business fields, such as surveillance videos in the security field, recording videos of sports and fitness, videos of cultural and film works, etc., and the single-frame image is an image of an object (i.e., a human) obtained by extracting frames from the video sequence.

[0074] Step 202: Perform clothing detection on the first image and mark the clothing detection boxes on the first image;

[0075] The first image is a single frame image of an object and clothing in a video sequence. Specifically, the first image is a single frame image in a video sequence, which includes, but is not limited to, videos in various existing business fields, such as surveillance videos in the security field, recording videos of sports and fitness, and videos of cultural and film works. The single frame image is an image of an object and clothing obtained by extracting frames from the video sequence.

[0076] Since the aforementioned clothing includes upper garments and / or lower garments, the clothing detection boxes marked on the first image are the clothing detection boxes for upper garments and / or lower garments. That is, in this embodiment, clothing matching of the object's upper garments and / or lower garments can be achieved.

[0077] Step 203: Take the object corresponding to each object detection box as the first object, and perform the operations of steps 204 to 205 for each first object:

[0078] The first image may include multiple different objects. Step 201 can draw the object detection box corresponding to each object. In this embodiment, each object is taken as the first object, and the clothing matching the first object is determined.

[0079] Step 204: Calculate the matching value between the target object detection box corresponding to the first object and each clothing detection box;

[0080] The matching value is used to characterize the proportion of the intersection area between the target object detection box and the clothing detection box within the target object detection box.

[0081] As described above, the formula for calculating the matching value is: Matching value = Intersection area of ​​the target object detection box and the clothing detection box / Target object detection box area; while the formula for calculating IOU is: Intersection area of ​​the clothing box and the human body box / Union area of ​​the clothing box and the human body box. The only difference between the matching value calculation method and the IOU calculation method is the denominator in the formula. When calculating the matching value, the denominator does not consider the target object detection box but only the clothing detection box. Therefore, when the clothing is incomplete, the calculated matching value will be larger than the IOU value. When setting the same preset threshold for comparison, it is not easy to filter out the incomplete clothing. Using the matching value in this embodiment for clothing-human body matching can effectively reduce the influence of the proportion of the target object detection box, i.e., the human body box, in the matching process and increase the influence of the clothing detection box, i.e., the clothing box, in the matching process, thereby improving the accuracy of clothing-human body matching.

[0082] Step 205: Determine the clothing that matches the first object based on multiple matching values.

[0083] After calculating the matching values ​​of the target object detection box and each clothing detection box corresponding to the first object through step 204 above, in order to determine which clothing matches the first object based on the matching values, it is necessary to compare the size of each matching value with the preset matching threshold; the clothing in the clothing detection box corresponding to the matching value greater than the preset matching threshold is determined as the clothing that matches the first object.

[0084] When the matching value is less than or equal to the preset matching threshold, it means that the clothing corresponding to the clothing detection box does not match the first object and is not the clothing worn by the first object. The preset matching threshold can be set according to actual needs and is not limited here.

[0085] The clothing matching method provided in this invention includes: performing object detection on a first image and marking object detection boxes for objects on the first image; performing clothing detection on the first image and marking clothing detection boxes for clothing on the first image; taking each object detection box as a first object, and performing the following operations for each first object: calculating the matching value between the target object detection box corresponding to the first object and each clothing detection box; and determining the clothing that matches the first object based on multiple matching values. In this invention, the matching value calculated based on the target object detection box and the clothing detection box is used to characterize the proportion of the intersection region of the target object detection box and the clothing detection box within the clothing detection box. In contrast, the traditional IOU calculation method characterizes the proportion of the intersection region within the union region. Compared to the IOU calculation method, since the denominator is modified to use the area of ​​the clothing detection box for calculation, the intersection score between the clothing detection box and the human body detection box will not change with changes in human body integrity. This effectively reduces the impact of human body integrity on the occlusion score (i.e., the matching value) during the matching process, improves the influence of the clothing detection box (i.e., the clothing box) during the matching process, and thus improves the accuracy of clothing-human body matching.

[0086] In some embodiments, step 201 above can be implemented through the following steps:

[0087] Step A1: Input the first image into the first neural network model to extract object location features from the first image using the first neural network model, and obtain the first feature map of the first image;

[0088] To meet the input requirements of the first neural network model, the first image can be scaled to a fixed size and then input into the first neural network model. The first neural network model can be trained based on deep learning networks such as convolutional neural networks and recurrent neural networks. The first neural network model is used to extract the positional features of objects in the first image to obtain the first feature map.

[0089] Step A2: Generate multiple first anchor boxes for each point on the first feature map, wherein each first anchor box has a different size;

[0090] Each generated first anchor frame has a different size and can be used to match human targets of different sizes. The number of first anchor frames can be set according to actual needs and is not limited here.

[0091] Step A3: For each first anchor box, calculate the first confidence that an object exists in the first anchor box;

[0092] Step A4: Retain the first target anchor box with a first confidence level greater than or equal to the first threshold, and determine the area enclosed by the first target anchor box as the object detection box of the object;

[0093] Specifically, a score (first confidence level) indicating the presence of a human body can be calculated in each first anchor box of the first feature map, thereby determining whether a human body exists in each first anchor box based on this score. A first anchor box containing a human body is a first target anchor box with a first confidence level greater than or equal to a first threshold. The area enclosed by the first target anchor box is the object detection box of the labeled object in the first image. This first threshold can be set according to actual needs and is not limited here.

[0094] To avoid duplicate detection when multiple first anchor boxes hit the same object, a non-maximum suppression algorithm can be used to select the best one from multiple first anchor boxes. Specifically, when there are multiple first target anchor boxes, the intersection-union ratio (IUR) of the multiple first target anchor boxes is determined; when the IUR is greater than or equal to a second threshold, the first target anchor box with the highest confidence is retained, and the area enclosed by the first target anchor box is determined as the object detection box.

[0095] First, the intersection-union ratio (IUR) of multiple first target anchor boxes is calculated. The IUR is used to identify which first target anchor boxes hit the same human body. The first target anchor boxes with an IUR greater than or equal to a second threshold are the first anchor boxes that hit the same human body. The second threshold can be set according to actual needs. Then, from the multiple first target anchor boxes that hit the same human body, the first target anchor box with the highest confidence is selected as the anchor box of the final human body region detection result, and the area enclosed by the first target anchor box is determined as the object detection box.

[0096] In addition to using the anchor-based bounding box method described in steps A1 to A4 above to annotate the object detection bounding box, other object detection methods can also be used, which are not limited here.

[0097] In some embodiments, step 202 above can be implemented through the following steps:

[0098] Step B1: Input the first image into the second neural network model to extract the position features of the lower garment in the first image using the second neural network model, and obtain the second feature map of the first image;

[0099] To meet the input requirements of the second neural network model, the first image can be scaled to a fixed size before being input into the second neural network model. The second neural network model can be trained based on deep learning networks such as convolutional neural networks and recurrent neural networks. The second neural network model is used to extract the positional features of the lower garment in the first image to obtain the second feature map.

[0100] Step B2: Generate multiple second anchor boxes for each point on the second feature map, wherein each second anchor box has a different size;

[0101] Step B3: For each second anchor frame, calculate the second confidence level that clothing exists in the second anchor frame;

[0102] Step B4: Retain the second target anchor box with a second confidence level greater than or equal to the first threshold, and determine the area enclosed by the second target anchor box as the clothing detection box.

[0103] Steps B1 to B4 are also based on the anchor box method to mark the garment detection box of the lower garment. The method of detecting the garment detection box is completely consistent with the method of detecting the object detection box mentioned above, and will not be described in detail here.

[0104] In some embodiments, step 204 above can be implemented through the following steps:

[0105] Step C1: In the image coordinate system of the first image, obtain the coordinate information of the first detection box of the target object detection box and the coordinate information of the second detection box of each clothing detection box;

[0106] Typically, the origin of the coordinate system for an image is set at the top left corner of the image. For ease of understanding, Figure 3 A schematic diagram of an image coordinate system for a first image is shown, such as... Figure 3 As shown, the origin O of the coordinate system is located at the upper left corner of the first image A, x represents the horizontal axis, and y represents the vertical axis.

[0107] In practical applications, the target object detection box and the clothing detection box can be determined using the coordinates of two diagonals. Further calculation of the matching value is then performed. The diagonal coordinates can be the coordinates of the top-left corner and the bottom-right corner, or the coordinates of the top-right corner and the bottom-left corner. For ease of explanation, this embodiment uses the coordinates of the top-left corner and the bottom-right corner of the two detection boxes as an example to illustrate the specific process of calculating the matching value. Figure 3 As shown, taking a target object detection box and a clothing detection box as examples, in the image coordinate system of the first image, the obtained coordinate information of the first detection box includes the first coordinate information (x1, y1) of the upper left corner position 1 of the target object detection box B and the second coordinate information (x2, y2) of the lower right corner position 2. The obtained coordinate information of the second detection box includes the third coordinate information (x3, y3) of the upper left corner position 3 of the clothing detection box C and the fourth coordinate information (x4, y4) of the lower right corner position 4.

[0108] Step C2: For each clothing detection frame, calculate the matching value based on the coordinate information of the first detection frame and the coordinate information of the second detection frame.

[0109] After obtaining the coordinate information of the target object detection box and the clothing detection box, the matching value can be calculated based on the coordinate information of the first detection box and the second detection box through steps D1 to D4:

[0110] Step D1: Determine the first area of ​​the clothing detection frame based on the third and fourth coordinate information;

[0111] The formula for calculating the first area of ​​the clothing detection frame is: Area1 = (x4 - x3) * (y4 - y3). The area occupied by the clothing detection frame can be accurately calculated using the above coordinates and calculation formula.

[0112] Step D2: Based on the first coordinate information, the second coordinate information, the third coordinate information, and the fourth coordinate information, determine the fifth coordinate information of the upper left corner and the sixth coordinate information of the lower right corner of the intersection area of ​​the target object detection box and the clothing detection box;

[0113] like Figure 3 As shown, the intersection of the target object detection box B and the clothing detection box C is the shaded area in the image. To determine the location of the intersection area in the first image, it is necessary to determine the fifth coordinate information of the upper left corner and the sixth coordinate information of the lower right corner of the intersection area. In actual use, the maximum value of the upper left corner and the minimum value of the lower right corner of the target object detection box and the clothing detection box are taken to determine the intersection area. Therefore, the fifth coordinate information of the upper left corner of the intersection area needs to be determined based on the first coordinate information and the third coordinate information; the sixth coordinate information of the lower right corner of the intersection area needs to be determined based on the second coordinate information and the fourth coordinate information.

[0114] Specifically, the fifth coordinate information for determining the intersection region can be described as follows: the largest horizontal coordinate is selected from the first horizontal coordinate of the first coordinate information and the third horizontal coordinate of the third coordinate information to determine the fifth horizontal coordinate of the fifth coordinate information; the largest vertical coordinate is selected from the first vertical coordinate of the first coordinate information and the third vertical coordinate of the third coordinate information to determine the fifth vertical coordinate of the fifth coordinate information.

[0115] As described above, the fifth horizontal axis coordinate x5 of the fifth coordinate information is max(x1, x3), meaning it is the maximum value between the horizontal axis coordinates x1 and x3 from the first and third coordinate information. Similarly, the fifth vertical axis coordinate y5 of the fifth coordinate information is max(y1, y3), meaning it is the maximum value between the vertical axis coordinates y1 and y3 from the first and third coordinate information. Figure 3 As can be seen from this, the fifth coordinate information of the upper left corner of the determined intersection area is the third coordinate information of the upper left corner of the clothing detection box.

[0116] Specifically, the sixth coordinate information for determining the intersection region can be described as follows: the smallest horizontal coordinate is selected from the second horizontal coordinate of the second coordinate information and the fourth horizontal coordinate of the fourth coordinate information to determine the sixth horizontal coordinate of the sixth coordinate information; the smallest vertical coordinate is selected from the second vertical coordinate of the second coordinate information and the fourth vertical coordinate of the fourth coordinate information to determine the sixth vertical coordinate of the sixth coordinate information.

[0117] As described above, the sixth horizontal axis coordinate x6 of the sixth coordinate information is min(x2, x4), meaning the maximum value between the horizontal axis coordinates x2 from the second coordinate information and x4 from the fourth coordinate information is taken as the horizontal axis coordinate of the sixth coordinate information. Similarly, the sixth vertical axis coordinate y6 of the sixth coordinate information is min(y2, y4), meaning the maximum value between the vertical axis coordinates y2 from the second coordinate information and y4 from the fourth coordinate information is taken as the vertical axis coordinate of the sixth coordinate information. Figure 3 As can be seen from this, the sixth coordinate information of the lower right corner of the determined intersection area is the second coordinate information of the lower right corner of the target object detection box.

[0118] Step D3: Determine the second area of ​​the intersection region based on the fifth and sixth coordinate information;

[0119] The formula for calculating the second area of ​​the intersection region is: Area2 = (x6 - x5) * (y6 - y5). Using the above coordinates and calculation formula, the area of ​​the region occupied by the intersection region can be accurately calculated.

[0120] Step D4: Calculate the matching value based on the first area and the second area.

[0121] The formula for calculating the matching value is M = Area2 / Area1. This formula can accurately determine the proportion of the intersection area within the clothing detection frame.

[0122] This embodiment also provides a keyframe extraction method, which can be executed by the aforementioned server 103, or jointly by server 103 and terminal 101, such as... Figure 4 As shown, the method may include the following steps:

[0123] Step 401: Obtain the target video sequence;

[0124] Step 402: Extract a single frame image from the target video sequence;

[0125] Step 403: Perform clothing matching on a single frame image to determine the matching clothing of the target object in the single frame image;

[0126] Step 404: Determine the single-frame image of the clothing that matches the target conditions as a keyframe;

[0127] The clothing matching method described above is included in the step of matching clothing to a single frame image.

[0128] In this embodiment, a video can be input, and frame extraction can be performed on the video. The extracted frames are then...

[0129] The obtained images are used for clothing matching to obtain matching clothing for the target objects in the video. Finally, based on actual business needs, matching clothing that meets certain conditions (e.g., clothing exposed in the frame exceeds a certain threshold) can be selected.

[0130] The keyframes for selecting the cover image should be the video frames containing the matching clothing (over 50%). If you need to choose an image from a short video as its cover, the cover image should be the most complete and unobstructed view of the target object's matching clothing. This will give you the keyframes required for the specific task.

[0131] 0 See Figure 5 This is a block diagram of an embodiment of a clothing matching device provided by the present invention;

[0132] like Figure 5 As shown, the device may include:

[0133] The first detection module 51 is used to perform object detection on the first image and mark the object detection box on the first image; wherein, the first image is a single frame image of the video sequence that includes the object;

[0134] The second detection module 52 is used to perform clothing detection on the first image and mark the clothing detection box on the first image.

[0135] Execution module 53 is used to take the object corresponding to each object detection box as the first object, and perform the following operations for each first object:

[0136] Calculation module 54 is used to calculate the target object detection box corresponding to the first object and the detection boxes of each garment.

[0137] The matching value of the detection box; where the matching value is used to characterize the proportion of the intersection area between the target object detection box and the clothing detection box in the clothing detection box;

[0138] The determination module 55 is used to determine the clothing that matches the first object based on multiple matching values.

[0139] The clothing matching device provided in this embodiment of the invention includes: performing object detection on a first image, marking object detection boxes on the first image, performing clothing detection on the first image, and performing clothing detection on the first image.

[0140] Clothing detection boxes are marked on an image; the object corresponding to each object detection box is designated as the first object, and the following operations are performed for each first object: calculate the target object corresponding to the first object.

[0141] The matching values ​​of the target object detection box and each clothing detection box are used to determine the clothing that matches the first object based on multiple matching values. In this invention, the matching value calculated based on the target object detection box and the clothing detection box is used to characterize the proportion of the intersection area of ​​the target object detection box and the clothing detection box in the clothing detection box. In contrast, the traditional IOU calculation method is used to characterize the proportion of the intersection area in the union area. Compared with the IOU calculation method, since the denominator is modified to use the area of ​​the clothing detection box for calculation, the intersection score of the clothing detection box and the human body detection box will not change with the change of human body integrity. This effectively reduces the influence of human body integrity on the occlusion score, i.e., the matching value, during the matching process, and improves the influence of the clothing detection box, i.e., the clothing box, during the matching process, thereby improving the accuracy of clothing-human body matching.

[0142] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 6 The illustrated electronic device 500 includes at least one processor 501, a memory 502, at least one network interface 504, and other user interfaces 503. The various components in the electronic device 500 are coupled together via a bus system 505. It is understood that the bus system 505 is used to implement communication between these components. In addition to a data bus, the bus system 505 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 6 The general designated all buses as Bus System 505.

[0143] The user interface 503 may include a display, keyboard, or clicking device (e.g., mouse, trackball, touchpad, or touchscreen).

[0144] It is understood that the memory 502 in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 502 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0145] In some implementations, memory 502 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 5021 and application program 5022.

[0146] The operating system 5021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 5022 includes various applications, such as a media player and a browser, used to implement various application functions. The program implementing the method of this embodiment can be included in the application program 5022.

[0147] In this embodiment of the invention, the processor 501 executes the method steps provided in each method embodiment by calling the program or instructions stored in the memory 502, specifically the program or instructions stored in the application program 5022.

[0148] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 501. Processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 501 or by instructions in the form of software. The processor 501 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 502. Processor 501 reads the information in memory 502 and, in conjunction with its hardware, completes the steps of the above method.

[0149] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0150] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0151] The electronic device provided in this embodiment may be as follows: Figure 6 The electronic device shown can perform the following: Figure 2 and 4 All steps of the method, thus achieving Figure 2 and 4 For details on the technical effects of the method shown, please refer to [link / reference]. Figure 2 and 4 The relevant descriptions are presented concisely and will not be elaborated upon here.

[0152] This invention also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; the memory may also include combinations of the above types of memory.

[0153] The above method can be implemented when one or more programs in the storage medium can be executed by one or more processors.

[0154] The processor is used to execute the aforementioned clothing matching and keyframe extraction programs stored in the memory to achieve... Figure 2 or Figure 4 The steps of the method shown.

[0155] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0156] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0157] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of matching apparel, comprising: The method includes: Performing object detection on a first image and marking object detection boxes on the first image includes: inputting the first image into a first neural network model to extract object location features from the first image using the first neural network model to obtain a first feature map of the first image; generating multiple first anchor boxes for each point on the first feature map, wherein each first anchor box has a different size; calculating a first confidence level for the presence of the object in each first anchor box; retaining first target anchor boxes with the first confidence level greater than or equal to a first threshold, and determining the region enclosed by the first target anchor boxes as the object detection box of the object; Clothing detection is performed on the first image, and clothing detection boxes are marked on the first image; wherein, the first image is a single frame image in a video sequence that includes objects and clothing, and the clothing includes upper clothing and / or lower clothing. Each object detection box corresponds to an object as a first object, and the following operations are performed for each first object: Calculate the matching value between the target object detection box corresponding to the first object and each of the clothing detection boxes; wherein, the matching value is used to characterize the proportion of the intersection area of ​​the target object detection box and the clothing detection box in the clothing detection box; The clothing that matches the first object is determined based on multiple matching values.

2. The method of claim 1, wherein, The step of performing clothing detection on the first image and marking clothing detection boxes on the first image includes: The first image is input into the second neural network model to extract clothing location features from the first image using the second neural network model, thereby obtaining the second feature map of the first image; For each point on the second feature map, generate multiple second anchor boxes, wherein each second anchor box has a different size; For each of the second anchor frames, calculate a second confidence level that the garment exists within the second anchor frame; The second target anchor frame with a confidence level greater than or equal to the first threshold is retained, and the area enclosed by the second target anchor frame is determined as the garment detection frame of the garment.

3. The method of claim 1, wherein, The calculation of the matching value between the target object detection box corresponding to the first object and each of the clothing detection boxes includes: In the image coordinate system of the first image, obtain the first detection box coordinate information of the target object detection box and the second detection box coordinate information of each of the clothing detection boxes; For each of the clothing detection frames, a matching value is calculated based on the coordinate information of the first detection frame and the coordinate information of the second detection frame.

4. The method of claim 3, wherein, The first detection box coordinate information includes the first coordinate information of the upper left corner position and the second coordinate information of the lower right corner position of the target object detection box; the second detection box coordinate information includes the third coordinate information of the upper left corner position and the fourth coordinate information of the lower right corner position of the clothing detection box. The calculation of the matching value based on the coordinate information of the first detection box and the coordinate information of the second detection box includes: The first area of ​​the clothing detection frame is determined based on the third coordinate information and the fourth coordinate information; Based on the first coordinate information, the second coordinate information, the third coordinate information, and the fourth coordinate information, determine the fifth coordinate information of the upper left corner and the sixth coordinate information of the lower right corner of the intersection area of ​​the target object detection box and the clothing detection box; The second area of ​​the intersection region is determined based on the fifth coordinate information and the sixth coordinate information; The matching value is calculated based on the first area and the second area.

5. The method of claim 4, wherein, The step of determining the fifth coordinate information of the upper left corner and the sixth coordinate information of the lower right corner of the intersection region based on the first coordinate information, the second coordinate information, the third coordinate information, and the fourth coordinate information includes: The fifth coordinate information is determined based on the first coordinate information and the third coordinate information; The sixth coordinate information is determined based on the second coordinate information and the fourth coordinate information.

6. The method of claim 5, wherein, Determining the fifth coordinate information based on the first coordinate information and the third coordinate information includes: The maximum horizontal axis coordinate is selected from the first horizontal axis coordinate of the first coordinate information and the third horizontal axis coordinate of the third coordinate information and determined as the fifth horizontal axis coordinate of the fifth coordinate information; The largest ordinate is selected from the first ordinate of the first coordinate information and the third ordinate of the third coordinate information and determined as the fifth ordinate of the fifth coordinate information.

7. The method of claim 5, wherein, Determining the sixth coordinate information based on the second coordinate information and the fourth coordinate information includes: The minimum horizontal axis coordinate is selected from the second horizontal axis coordinate of the second coordinate information and the fourth horizontal axis coordinate of the fourth coordinate information and determined as the sixth horizontal axis coordinate of the sixth coordinate information; The minimum vertical coordinate is selected from the second vertical coordinate of the second coordinate information and the fourth vertical coordinate of the fourth coordinate information to determine the sixth vertical coordinate of the sixth coordinate information.

8. The method of claim 1, wherein, The step of determining the clothing that matches the first object based on multiple matching values ​​includes: Compare each of the matching values ​​with a preset matching threshold; The clothing in the clothing detection box corresponding to a matching value greater than the preset matching threshold is determined as clothing that matches the first object.

9. A key frame extraction method characterized by, The method includes: Obtain the target video sequence; Extract a single frame image from the target video sequence; Perform clothing matching on the single frame image to determine the matching clothing of the target object in the single frame image; The single-frame image in which the matched clothing meets the target conditions is determined as a keyframe; The step of matching clothing to the single frame image includes the clothing matching method described in any one of claims 1 to 8.

10. A garment matching device, characterized by, The device includes: The first detection module is used to perform object detection on the first image and mark object detection boxes on the first image. Specifically, it is used to: input the first image into a first neural network model to extract object location features from the first image using the first neural network model to obtain a first feature map of the first image; generate multiple first anchor boxes for each point on the first feature map, wherein each first anchor box has a different size; calculate a first confidence level for the presence of the object in each first anchor box; retain first target anchor boxes with the first confidence level greater than or equal to a first threshold, and determine the area enclosed by the first target anchor boxes as the object detection box of the object; The second detection module is used to detect clothing in the first image and mark the clothing detection box on the first image; wherein, the first image is a single frame image in a video sequence that includes objects and clothing, and the clothing includes upper clothing and / or lower clothing. The execution module is used to treat the object corresponding to each object detection box as a first object, and perform the following operations for each first object: The calculation module is used to calculate the matching value between the target object detection box corresponding to the first object and each of the clothing detection boxes; wherein, the matching value is used to characterize the proportion of the intersection area of ​​the target object detection box and the clothing detection box in the clothing detection box; The determining module is used to determine the clothing that matches the first object based on a plurality of the matching values.

11. An electronic device, comprising: include: A processor and a memory, the processor being configured to execute a clothing matching and keyframe extraction program stored in the memory to implement the steps of the method according to any one of claims 1 to 8 or 9.

12. A storage medium, characterized by The storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the method according to any one of claims 1 to 8 or 9.

Images