Clothing matching methods, devices, electronic devices and storage media

Abstract

The clothing matching method, apparatus, electronic device, and storage medium provided in this invention include: performing object detection and clothing detection on a first image; marking object detection boxes for objects and clothing detection boxes for clothing on the first image; searching for a first clothing detection box that matches the target object detection box corresponding to the first object; when there are multiple first clothing detection boxes corresponding to upper or lower garments, determining a first human body key point of the first object based on a second image of the image region where the target object detection box is located; determining a target clothing detection box from the multiple first clothing detection boxes based on the first human body key point, so as to match the clothing corresponding to the target clothing detection box with the first object. This invention can accurately determine the clothing that truly matches the object from multiple matching clothing detection boxes based on the human body key points corresponding to the object, improving the accuracy of clothing and corresponding human body matching in complex scenes.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, apparatus, electronic device and storage medium for matching clothing. Background Technology

[0002] In video scenes, by analyzing the type and style of clothing worn by characters, we can gain a deeper understanding of their image and personality, which can help recommend suitable products to them or help analyze their positioning in the whole drama and the direction of the story. As a prerequisite for analyzing characters through clothing, it is necessary to match the characters with the corresponding clothing.

[0003] Currently, in order to obtain the matching relationship between clothing and people, AI (Artificial Intelligence) models can be used to detect clothing boxes and human body boxes in the video. Based on this, the IOU (Intersection over Union) or other ratio scores between clothing boxes and human body boxes can be calculated. A threshold can be set based on this score and all matching pairs above the threshold can be retained, or other post-processing strategies can be used to finally obtain clothing that matches the human body.

[0004] However, due to the complexity of video scenes, when two people are very close, using the IOU method or similar methods can easily lead to matching errors. For example, matching the clothing worn by multiple different people to the same person reduces the accuracy of clothing-body matching. Summary of the Invention

[0005] In view of this, in order to solve the above-mentioned technical problems or some of the technical problems, the embodiments of the present invention provide a clothing matching method, device, electronic device and storage medium, which can improve the accuracy of clothing-human body matching in complex scenarios.

[0006] In a first aspect, embodiments of the present invention provide a clothing matching method, wherein the method includes:

[0007] Object detection and clothing detection are performed on the first image, and the object detection bounding box of the object and the clothing detection bounding box of the clothing are marked on the first image; wherein, the first image is a single frame image in the video sequence that includes objects and clothing, and the clothing includes upper clothing and / or lower clothing.

[0008] For each object detection box, the object corresponding to that object is designated as the first object. For each first object, the following operations are performed:

[0009] Find the first clothing detection box that matches the target object detection box corresponding to the first object;

[0010] When there are multiple first clothing detection boxes corresponding to the upper or lower clothing, the second image of the image area where the target object detection box is located is captured, and the first human body key point of the first object is determined based on the second image.

[0011] Based on the first human body key points, a target clothing detection box is determined from multiple first clothing detection boxes, so as to match the clothing corresponding to the target clothing detection box with the first object.

[0012] In one possible implementation, object detection and clothing detection are performed on the first image, and object detection bounding boxes for objects and clothing detection bounding boxes for clothing are marked on the first image, including:

[0013] The first image is input into the first neural network model to extract object location features from the first image using the first neural network model, thereby obtaining the first feature map of the first image;

[0014] For each point on the first feature map, generate multiple first anchor boxes, where each first anchor box has a different size;

[0015] For each first anchor box, calculate the first confidence that an object exists in the first anchor box;

[0016] The first target anchor box with a first confidence level greater than or equal to the first threshold is retained, and the region enclosed by the first target anchor box is determined as the object detection box of the object.

[0017] The first image is input into the second neural network model to extract clothing location features from the first image using the second neural network model, thereby obtaining the second feature map of the first image;

[0018] For each point on the second feature map, generate multiple second anchor boxes, where each second anchor box has a different size;

[0019] For each second anchor box, calculate the second confidence level that clothing exists in the second anchor box;

[0020] The second target anchor box with a second confidence level greater than or equal to the first threshold is retained, and the area enclosed by the second target anchor box is determined as the clothing detection box.

[0021] In one possible implementation, finding a first clothing detection box that matches the target object detection box corresponding to the first object includes:

[0022] Calculate the IOU value between the target object detection box and each clothing detection box;

[0023] The clothing detection box corresponding to the largest IOU value is used as the first clothing detection box.

[0024] In one possible implementation, determining the first human body key points of the first object based on the second image includes:

[0025] The second image is input into the third neural network model to extract human features from the second image, thereby obtaining the third feature map of the second image.

[0026] The third feature map is converted into multiple heat maps, where each heat map corresponds to a preset human body key point;

[0027] Determine the thermal value of each point on each heatmap, and sort each point on the heatmap according to the thermal value;

[0028] The coordinates of the points before the target sorting position are weighted and averaged to obtain the coordinate information of the predicted human key point corresponding to each preset human key point. The first human key point includes the coordinate information of the predicted human key point.

[0029] Based on the coordinate information, predict the key points of the human body in the second image.

[0030] In one possible implementation, a target clothing detection box is determined from a plurality of first clothing detection boxes based on first human body key points, including:

[0031] Count the number of key points including the first human body key point in each first clothing detection frame;

[0032] The first clothing detection box corresponding to the maximum number of key points is determined as the target clothing detection box.

[0033] In one possible implementation, the first human body key point carries a first object identifier of the first object;

[0034] Count the number of keypoints including the first human body keypoint in each first clothing detection frame, including:

[0035] For each first clothing detection frame, find the first human body key point carrying the first object identifier within the first clothing detection frame;

[0036] Count the number of key points in the first human body.

[0037] Secondly, embodiments of the present invention provide a keyframe extraction method, wherein the method includes:

[0038] Obtain the target video sequence;

[0039] Extract single-frame images from the target video sequence;

[0040] Perform clothing matching on a single frame image to determine the matching clothing of the target object in the single frame image;

[0041] The single-frame image that matches the clothing and meets the target conditions is determined as the keyframe;

[0042] The clothing matching method described above is included in the step of matching clothing to a single frame image.

[0043] Thirdly, embodiments of the present invention provide a clothing matching device, wherein the device includes:

[0044] The detection module is used to perform object detection and clothing detection on the first image, and to mark the object detection box of the object and the clothing detection box of the clothing on the first image; wherein, the first image is a single frame image in the video sequence that includes objects and clothing, and the clothing includes upper clothing and / or lower clothing.

[0045] The execution module is used to treat the object corresponding to each object detection box as the first object, and perform the following operations for each first object:

[0046] The search module is used to find the first clothing detection box that matches the target object detection box corresponding to the first object;

[0047] The first determining module is used to extract a second image of the image area where the target object detection box is located when there are multiple first clothing detection boxes corresponding to the upper or lower clothing, and determine the first human body key points of the first object based on the second image.

[0048] The second determining module is used to determine a target clothing detection box from multiple first clothing detection boxes based on the first human body key points, so as to match the clothing corresponding to the target clothing detection box with the first object.

[0049] Fourthly, embodiments of the present invention provide an electronic device, comprising: a processor and a memory, wherein the processor is configured to execute a clothing matching and keyframe extraction program stored in the memory to implement the steps of the above-described method.

[0050] Fifthly, embodiments of the present invention provide a storage medium, wherein the storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the above-described method.

[0051] The clothing matching method, apparatus, electronic device, and storage medium provided in this invention include: performing object detection and clothing detection on a first image; marking object detection boxes for objects and clothing detection boxes for clothing on the first image; designating each object detection box as a first object; searching for a first clothing detection box that matches the target object detection box corresponding to the first object; when there are multiple first clothing detection boxes corresponding to upper or lower garments, cropping a second image of the image area where the target object detection box is located; determining a first human body key point of the first object based on the second image; and determining a target clothing detection box from the multiple first clothing detection boxes based on the first human body key point, so as to match the clothing corresponding to the target clothing detection box with the first object. Research has shown that the clothing detection box corresponding to the clothing matched with the object should include the human body key point of the object with the most objects. Therefore, in cases where an object is mismatched with multiple upper or lower garments, this invention can accurately determine the clothing that truly matches the object from multiple matching clothing detection boxes based on the human body key point corresponding to the object, effectively solving the problem of mismatched human bodies and clothing in dense crowds in complex scenes, and improving the accuracy of clothing and corresponding human body matching in complex scenes. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the hardware environment for a clothing matching method provided in an embodiment of the present invention;

[0053] Figure 2 A schematic flowchart illustrating a clothing matching method provided in an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram of the distribution of preset human body key points provided in an embodiment of the present invention;

[0055] Figure 4 A flowchart illustrating a keyframe extraction method provided in an embodiment of the present invention;

[0056] Figure 5 A block diagram illustrating an embodiment of a clothing matching device provided by the present invention;

[0057] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] To facilitate understanding of the embodiments of the present invention, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0060] In this embodiment, the above-described clothing matching method can be applied to, for example... Figure 1 The hardware environment shown consists of terminal 101 and server 103. Figure 1 As shown, server 103 is connected to terminal 101 via a network and can be used to provide services (such as clothing matching services) to the terminal or clients installed on the terminal. Database 105 can be set up on the server or independently of the server to provide data storage services for server 103. The network mentioned above includes, but is not limited to, wide area network, metropolitan area network or local area network. Terminal 101 includes, but is not limited to, PC, mobile phone, tablet computer, etc.

[0061] The clothing matching method in this embodiment can be executed by server 103, or it can be executed jointly by server 103 and terminal 101, such as... Figure 2 As shown, the method may include the following steps:

[0062] Step 201: Perform object detection and clothing detection on the first image, and mark the object detection bounding box of the object and the clothing detection bounding box of the clothing on the first image.

[0063] The first image is a single frame image of an object and clothing in a video sequence; specifically, the first image is a single frame image in a video sequence, which includes, but is not limited to, videos in various existing business areas, such as surveillance videos in the security field, recording videos of sports and fitness, and videos of cultural and film works.

[0064] Since the aforementioned clothing includes upper garments and / or lower garments, the clothing detection boxes marked on the first image are the clothing detection boxes for upper garments and / or lower garments. That is, in this embodiment, clothing matching of the object's upper garments and / or lower garments can be achieved.

[0065] Step 202: Take the object corresponding to each object detection box as the first object, and perform the operations from step 203 to step 205 for each first object:

[0066] The first image may include multiple different objects. Step 201 can draw the object detection box corresponding to each object. In this embodiment, each object is taken as the first object, and the clothing matching the first object is determined.

[0067] Step 203: Find the first clothing detection box that matches the target object detection box corresponding to the first object;

[0068] Since there are multiple objects in the first image, there are multiple garments. Step 201 can draw multiple garment detection boxes. In order to match the same first object with clothing, it is first necessary to find the first garment detection box that matches the target object detection box of the first object from the multiple garment detection boxes. The garments defined in the first garment detection box can be upper garments or lower garments.

[0069] If step 202 determines that the target object detection boxes of the first object all match a first clothing detection box of an upper garment and / or a first clothing detection box of a lower garment, then obviously, the clothing matched by the first object is correct, and therefore, steps 204-205 do not need to be executed.

[0070] Step 204: When there are multiple first clothing detection boxes corresponding to the upper or lower clothing, capture the second image of the image area where the target object detection box is located, and determine the first human body key point of the first object based on the second image.

[0071] If the target object detection box matches multiple first clothing detection boxes corresponding to top or bottom clothing, it means that the first object has matched clothing worn by other objects. Therefore, steps 204 to 205 need to be executed to determine the unique clothing that matches the first object from the multiple matched first clothing detection boxes.

[0072] Step 205: Based on the first human body key points, determine the target clothing detection box from multiple first clothing detection boxes, so as to match the clothing corresponding to the target clothing detection box with the first object.

[0073] Studies have shown that the clothing detection box corresponding to clothing that matches the object overlaps more with the object detection box, thus the clothing detection box contains more human body key points of the object. On the other hand, the clothing detection box corresponding to clothing that does not match the object overlaps less with the object detection box, thus the clothing detection box contains few or no human body key points of the object. Based on the above research, in this embodiment, the clothing in the first clothing detection box that contains the most first human body key points among the multiple first clothing detection boxes that match the target object detection box is taken as the matching clothing of the first object. Therefore, in this step 205, the target clothing detection box is determined from the multiple first clothing detection boxes based on the first human body key points, so as to match the clothing corresponding to the target clothing detection box with the first object.

[0074] In one embodiment, the specific implementation of determining the target clothing detection frame from multiple first clothing detection frames based on the first human body key points may include: counting the number of first human body key points in each first clothing detection frame.

[0075] The number of key points of the body key points; the first clothing detection box corresponding to the maximum number of key points is determined as the target clothing detection box.

[0076] To indicate which specific human body key point was detected, step 204 yields the...

[0077] Each of the first key points on the human body of an object carries a first object identifier, which can uniquely identify the aforementioned first object. Therefore, the first object identifier of each first clothing detection frame is statistically analyzed.

[0078] The specific process for determining the number of key points of a human body is as follows: find the first human body key point carrying the first object identifier in the first clothing detection box; count the number of key points of the first human body key point.

[0079] In actual use, if the first pair is not found in the first clothing detection box...

[0080] If the first human body keypoint is identified, then the keypoint data for the first human body keypoint is 0 or 5. This indicates that the target object detection box of the first object and the first clothing detection box are mismatched.

[0081] The clothing in the clothing detection box refers to the clothing worn by other objects, not the clothing worn by the first object.

[0082] Suppose there are 3 first clothing detection boxes that match the target object detection box. Among them, the first clothing detection box 1 has 0 key points for the first human body key point.

[0083] The first human body keypoint count in detection frame 2 is 3, and the first clothing keypoint count in detection frame 3 is 10. Therefore, the first clothing keypoint count in detection frame 3 can be...

[0084] The corresponding clothing is matched with the first object, that is, the clothing corresponding to the first clothing detection box 3 is determined to be the clothing worn by the first object.

[0085] By counting the number of key points of the first human body in each first clothing detection frame,

[0086] The clothing in the first clothing detection frame with the largest number of key points, as the first object's clothing, can be matched using the clothing 5 matching method. This method can determine the clothing that matches the first object based on the quantification of human body key points, which is convenient for root...

[0087] We conduct analysis and parameter tuning based on actual business scenarios. For example, we select the clothing image corresponding to the clothing detection box that best matches the first object, so as to accurately match it with Taobao links.

[0088] The clothing matching method provided in this invention includes: performing object detection and clothing detection on a first image; marking object detection boxes for objects and clothing detection boxes for clothing on the first image; designating each object detection box as a first object; finding a first clothing detection box that matches the target object detection box corresponding to the first object; when there are multiple first clothing detection boxes corresponding to upper or lower garments, cropping a second image of the image region where the target object detection box is located; determining a first human body key point of the first object based on the second image; and determining a target clothing detection box from the multiple first clothing detection boxes based on the first human body key point, so as to match the clothing corresponding to the target clothing detection box with the first object. Research has shown that the clothing detection box corresponding to the clothing matched with the object should include the human body key point of the object with the most objects. Therefore, in cases where an object is mismatched with multiple upper or lower garments, this invention can accurately determine the clothing that truly matches the object from multiple matching clothing detection boxes based on the human body key point corresponding to the object, effectively solving the problem of mismatched human bodies and clothing in dense crowds in complex scenes, and improving the accuracy of clothing and corresponding human body matching in complex scenes.

[0089] In some embodiments, step 201 above can be implemented through the following steps:

[0090] Step A1: Input the first image into the first neural network model to extract object location features from the first image using the first neural network model, and obtain the first feature map of the first image;

[0091] To meet the input requirements of the first neural network model, the first image can be scaled to a fixed size and then input into the first neural network model. The first neural network model can be trained based on deep learning networks such as convolutional neural networks and recurrent neural networks. The first neural network model is used to extract the positional features of objects in the first image to obtain the first feature map.

[0092] Step A2: Generate multiple first anchor boxes for each point on the first feature map, wherein each first anchor box has a different size;

[0093] Each generated first anchor frame has a different size and can be used to match human targets of different sizes. The number of first anchor frames can be set according to actual needs and is not limited here.

[0094] Step A3: For each first anchor box, calculate the first confidence level that an object exists in the first anchor box;

[0095] Step A4: Retain the first target anchor box with a first confidence level greater than or equal to the first threshold, and determine the area enclosed by the first target anchor box as the object detection box of the object;

[0096] Specifically, a score (first confidence level) indicating the presence of a human body can be calculated in each first anchor box of the first feature map, thereby determining whether a human body exists in each first anchor box based on this score. A first anchor box containing a human body is a first target anchor box with a first confidence level greater than or equal to a first threshold. The area enclosed by the first target anchor box is the object detection box of the labeled object in the first image. This first threshold can be set according to actual needs and is not limited here.

[0097] To avoid duplicate detection when multiple first anchor boxes hit the same object, a non-maximum suppression algorithm can be used to select the best one from multiple first anchor boxes. Specifically, when there are multiple first target anchor boxes, the intersection-union ratio (IUR) of the multiple first target anchor boxes is determined; when the IUR is greater than or equal to a second threshold, the first target anchor box with the highest confidence is retained, and the area enclosed by the first target anchor box is determined as the object detection box.

[0098] First, the intersection-union ratio (IUR) of multiple first target anchor boxes is calculated. The IUR is used to identify which first target anchor boxes hit the same human body. The first target anchor boxes with an IUR greater than or equal to a second threshold are the first anchor boxes that hit the same human body. The second threshold can be set according to actual needs. Then, from the multiple first target anchor boxes that hit the same human body, the first target anchor box with the highest confidence is selected as the anchor box of the final human body region detection result, and the area enclosed by the first target anchor box is determined as the object detection box.

[0099] In addition to using the anchor-based bounding box method described in steps A1 to A4 above to annotate the object detection bounding box, other object detection methods can also be used, which are not limited here.

[0100] Step A6: Input the first image into the second neural network model to extract clothing location features from the first image using the second neural network model, and obtain the second feature map of the first image;

[0101] To meet the input requirements of the second neural network model, the first image can be scaled to a fixed size before being input into the second neural network model. The second neural network model can be trained based on deep learning networks such as convolutional neural networks and recurrent neural networks. The second neural network model is used to extract the positional features of clothing in the first image to obtain the second feature map.

[0102] Step A7: Generate multiple second anchor boxes for each point on the second feature map, wherein each second anchor box has a different size;

[0103] Step A8: For each second anchor box, calculate the second confidence level that clothing exists in the second anchor box;

[0104] Step A9: Retain the second target anchor box with a second confidence level greater than or equal to the first threshold, and determine the area enclosed by the second target anchor box as the clothing detection box.

[0105] Steps A6 to A9 are also the process of marking the clothing detection box based on the anchor box method. The method of detecting the clothing detection box is completely consistent with the method of detecting the object detection box mentioned above, and will not be described in detail here.

[0106] In some embodiments, step 203 above can be implemented through the following steps:

[0107] Step B1: Calculate the IOU value between the target object detection box and each clothing detection box;

[0108] In this embodiment, the Intersection over Union (IOU) matching tracking algorithm can be used to match the target object detection box with each clothing detection box to determine the clothing corresponding to the same first object. For example, clothing detection boxes 1, 2, and 3 are detected in the first image. When performing detection box matching, the IOU matching tracking algorithm can be used to calculate the IOU values ​​between the target object detection box and the three clothing detection boxes. The IOU value between the target object detection box and clothing detection box 1 is 0.4, the IOU value between the target object detection box and clothing detection box 2 is 0.6, and the IOU value between the target object detection box and clothing detection box 3 is 0.9.

[0109] Step B2: Use the clothing detection box corresponding to the maximum IOU value as the first clothing detection box.

[0110] Continuing from the previous example, since the IOU value between the target object detection box and the clothing detection box 3 is the highest, it can be determined that the clothing detection box 3 is the first clothing detection box corresponding to the target object detection box.

[0111] The embodiments of the present invention can accurately obtain the first clothing detection box that matches the target object detection box using the IOU matching tracking algorithm, and realize clothing matching detection of the same first object based on the target object detection box and the first clothing detection box, thereby improving the accuracy of clothing matching.

[0112] In some embodiments, step 204 above can be implemented through the following steps:

[0113] Step C1: Input the second image into the third neural network model to extract human features from the second image using the third neural network model, and obtain the third feature map of the second image;

[0114] To meet the input requirements of the third neural network model, the second image can be scaled to a fixed size before being input into the third neural network model. The third neural network model can be trained based on deep learning networks such as convolutional neural networks and recurrent neural networks. The third neural network model is used to extract the human features of the first object in the second image to obtain the third feature map.

[0115] Step C2: Convert the third feature map into multiple heatmaps, where each heatmap corresponds to a preset human body key point;

[0116] Step C3: Determine the thermal value of each point on each heat map, and sort each point on the heat map according to the thermal value;

[0117] like Figure 3 As shown, the actual position of multiple preset human body key points can be predicted. These preset human body key points include key points of the human body parts such as the top of the head, left ear, right ear, left eye, right eye, nose, left shoulder, right shoulder, left elbow, left wrist, right elbow, right wrist, left hip, right hip, left knee, left ankle, right knee, right ankle, left chest, right chest, and navel.

[0118] The number of heatmaps is the same as the number of preset human body key points and they correspond one-to-one. The heat value of each point on each heatmap is the probability value that the point belongs to the corresponding preset human body key point. This probability value is calculated by the second neural network model.

[0119] Step C4: Take a weighted average of the coordinates of the points before the target sorting position to obtain the coordinate information of the predicted human key point corresponding to each preset human key point. The first human key point includes the coordinate information of the predicted human key point.

[0120] The target's ranking position can be among the top 2, top 3, etc., and can be set according to actual needs. By taking a weighted average of the coordinates of the points preceding the target's ranking position, the actual position of the predicted human keypoint corresponding to the preset human keypoint can be obtained.

[0121] Step C5: Draw the predicted human body key points in the second image according to the coordinate information.

[0122] The predicted human body key points are the first human body key points of the first object determined based on the second image. The determined first human body key points include one or more of the following: top of head, left ear, right ear, left eye, right eye, nose, left shoulder, right shoulder, left elbow, left wrist, right elbow, right wrist, left hip, right hip, left knee, left ankle, right knee, right ankle, left chest, right chest, and navel.

[0123] In this embodiment, in addition to the heatmap-based human key point detection method in step 204 above, other human key point detection methods can also be used, and no limitation is made here.

[0124] This embodiment also provides a keyframe extraction method, which can be executed by the aforementioned server 103, or jointly by server 103 and terminal 101, such as... Figure 4 As shown, the method may include the following steps:

[0125] Step 401: Obtain the target video sequence;

[0126] Step 402: Extract a single frame image from the target video sequence;

[0127] Step 403: Perform clothing matching on a single frame image to determine the matching clothing of the target object in the single frame image;

[0128] Step 404: Determine the single-frame image of the clothing that matches the target conditions as a keyframe;

[0129] The clothing matching method described above is included in the step of matching clothing to a single frame image.

[0130] In this embodiment, a video can be input, and frame extraction can be performed. The resulting images are then used for clothing matching to obtain the matching clothing for the target object in the video. Finally, based on actual business needs, the video frames containing the matching clothing that meet certain conditions (e.g., the matching clothing is exposed in more than 50% of the frame) can be selected. For example, if an image needs to be selected from a short video as its cover, the most complete and unobstructed image of the target object's matching clothing should be used as the cover, thus obtaining the keyframe required for the actual business.

[0131] See Figure 5 This is a block diagram illustrating an embodiment of a clothing matching device provided by the present invention; as shown below. Figure 5 As shown, the device may include:

[0132] The detection module 51 is used to perform object detection and clothing detection on the first image, and to mark the object detection box of the object and the clothing detection box of the clothing on the first image; wherein, the first image is a single frame image in the video sequence that includes objects and clothing, and the clothing includes upper clothing and / or lower clothing.

[0133] Execution module 52 is used to take the object corresponding to each object detection box as the first object, and perform the following operations for each first object:

[0134] The search module 53 is used to search for a first clothing detection box that matches the target object detection box corresponding to the first object;

[0135] The first determining module 54 is used to capture a second image of the image area where the target object detection box is located when there are multiple first clothing detection boxes corresponding to the upper or lower clothing, and determine the first human body key points of the first object based on the second image.

[0136] The second determining module 55 is used to determine a target clothing detection box from multiple first clothing detection boxes based on the first human body key points, so as to match the clothing corresponding to the target clothing detection box with the first object.

[0137] The clothing matching device provided in this invention includes: performing object detection and clothing detection on a first image; marking object detection boxes for objects and clothing detection boxes for clothing on the first image; designating each object detection box as a first object; searching for a first clothing detection box that matches the target object detection box corresponding to the first object; when there are multiple first clothing detection boxes corresponding to upper or lower garments, cropping a second image of the image area where the target object detection box is located; determining a first human body key point of the first object based on the second image; and determining a target clothing detection box from the multiple first clothing detection boxes based on the first human body key point, so as to match the clothing corresponding to the target clothing detection box with the first object. Research has shown that the clothing detection box corresponding to the clothing matched with the object should include the human body key point of the object with the most objects. Therefore, in the case of mismatched objects with multiple upper or lower garments, this invention can accurately determine the clothing that truly matches the object from multiple matching clothing detection boxes based on the human body key points corresponding to the object, effectively solving the problem of mismatched human bodies and clothing in dense crowds in complex scenes, and improving the accuracy of clothing and corresponding human body matching in complex scenes.

[0138] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 6 The illustrated electronic device 500 includes at least one processor 501, a memory 502, at least one network interface 504, and other user interfaces 503. The various components in the electronic device 500 are coupled together via a bus system 505. It is understood that the bus system 505 is used to implement communication between these components. In addition to a data bus, the bus system 505 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 6 The general designated all buses as Bus System 505.

[0139] The user interface 503 may include a display, keyboard, or clicking device (e.g., mouse, trackball, touchpad, or touchscreen).

[0140] It is understood that the memory 502 in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (OEPROM), electrically erasable programmable read-only memory (EEPROM), or...

[0141] Flash memory. Volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (SRAM), etc.

[0142] Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 502 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0143] In some implementations, memory 502 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 5021 and application program 5022.

[0144] The operating system 5021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 5022 includes various applications, such as a media player and a browser, used to implement various application functions. The program implementing the method of this embodiment can be included in the application program 5022.

[0145] In this embodiment of the invention, the processor 501 executes the method steps provided in each method embodiment by calling the program or instructions stored in the memory 502, specifically the program or instructions stored in the application program 5022.

[0146] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 501. Processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 501 or by instructions in the form of software. The processor 501 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 502. Processor 501 reads the information in memory 502 and, in conjunction with its hardware, completes the steps of the above method.

[0147] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0148] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0149] The electronic device provided in this embodiment may be as follows: Figure 6 The electronic device shown can perform the following: Figure 2 and 4 All steps of the method, thus achieving Figure 2 and 4 For details on the technical effects of the method shown, please refer to [link / reference]. Figure 2 and 4 The relevant descriptions are presented concisely and will not be elaborated upon here.

[0150] This invention also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; the memory may also include combinations of the above types of memory.

[0151] The above method can be implemented when one or more programs in the storage medium can be executed by one or more processors.

[0152] The processor is used to execute clothing matching and keyframe extraction programs stored in the memory to achieve... Figure 2 or Figure 4 The steps of the method shown.

[0153] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0154] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0155] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for matching clothing, characterized in that, The method includes: Object detection and clothing detection are performed on the first image, and object detection boxes and clothing detection boxes are marked on the first image; wherein, the first image is a single frame image in a video sequence that includes objects and clothing, and the clothing includes upper clothing and / or lower clothing. Each object detection box corresponds to an object as a first object, and the following operations are performed for each first object: Find the first clothing detection box that matches the target object detection box corresponding to the first object; When there are multiple first clothing detection frames corresponding to the upper garment or the lower garment, a second image of the image area where the target object detection frame is located is captured, and the first human body key point of the first object is determined based on the second image. Based on the first human body key points, a target clothing detection box is determined from multiple first clothing detection boxes to match the clothing corresponding to the target clothing detection box with the first object. This includes: counting the number of key points in each first clothing detection box that include the first human body key points; and determining the first clothing detection box corresponding to the largest number of key points as the target clothing detection box.

2. The method according to claim 1, characterized in that, The step of performing object detection and clothing detection on the first image, and marking the object detection bounding box of the object and the clothing detection bounding box of the clothing on the first image, includes: The first image is input into the first neural network model to extract object location features from the first image using the first neural network model, thereby obtaining the first feature map of the first image; For each point on the first feature map, generate multiple first anchor boxes, wherein each first anchor box has a different size; For each of the first anchor frames, calculate a first confidence level that the object exists in the first anchor frame; The first target anchor box with a confidence level greater than or equal to the first threshold is retained, and the area enclosed by the first target anchor box is determined as the object detection box of the object. The first image is input into the second neural network model to extract clothing location features from the first image using the second neural network model, thereby obtaining the second feature map of the first image; For each point on the second feature map, generate multiple second anchor boxes, wherein each second anchor box has a different size; For each of the second anchor frames, calculate a second confidence level that the garment exists within the second anchor frame; The second target anchor frame with a confidence level greater than or equal to the first threshold is retained, and the area enclosed by the second target anchor frame is determined as the garment detection frame of the garment.

3. The method according to claim 1, characterized in that, The step of finding the first clothing detection box that matches the target object detection box corresponding to the first object includes: Calculate the IOU value between the target object detection box and each of the clothing detection boxes; The clothing detection box corresponding to the largest IOU value is used as the first clothing detection box.

4. The method according to claim 1, characterized in that, Determining the first human key points of the first object based on the second image includes: The second image is input into the third neural network model to extract human features from the second image using the third neural network model, thereby obtaining the third feature map of the second image; The third feature map is converted into multiple heat maps, where each heat map corresponds to a preset human body key point; In each of the heatmaps, a thermal value is determined for each point on the heatmap, and each point on the heatmap is sorted according to the thermal value. The coordinates of the points before the target sorting position are weighted and averaged to obtain the coordinate information of the predicted human key point corresponding to each preset human key point. The first human key point includes the coordinate information of the predicted human key point. The predicted human body key points are drawn in the second image according to the coordinate information.

5. The method according to claim 1, characterized in that, The first human body key point carries the first object identifier of the first object; The counting of key points including the first human body key points in each of the first clothing detection frames includes: For each of the first clothing detection frames, find the first human body key point carrying the first object identifier in the first clothing detection frame; Count the number of key points of the first human body.

6. A keyframe extraction method, characterized in that, The method includes: Obtain the target video sequence; Extract a single frame image from the target video sequence; Perform clothing matching on the single frame image to determine the matching clothing of the target object in the single frame image; The single-frame image in which the matching clothing meets the target conditions is determined as a keyframe; The step of matching clothing to the single frame image includes the clothing matching method described in any one of claims 1 to 5.

7. A clothing matching device, characterized in that, The device includes: The detection module is used to perform object detection and clothing detection on the first image, and to mark the object detection box of the object and the clothing detection box of the clothing on the first image; wherein, the first image is a single frame image in a video sequence that includes objects and clothing, and the clothing includes upper clothing and / or lower clothing. The execution module is used to treat the object corresponding to each object detection box as a first object, and perform the following operations for each first object: The search module is used to find a first clothing detection box that matches the target object detection box corresponding to the first object; The first determining module is used to, when there are multiple first clothing detection frames corresponding to the upper clothing or the lower clothing, capture a second image of the image area where the target object detection frame is located, and determine the first human body key point of the first object based on the second image; The second determining module is used to determine a target clothing detection box from multiple first clothing detection boxes based on the first human body key points, so as to match the clothing corresponding to the target clothing detection box with the first object. Specifically, it is used to: count the number of key points including the first human body key points in each first clothing detection box; and determine the first clothing detection box corresponding to the largest number of key points as the target clothing detection box.

8. An electronic device, characterized in that, include: A processor and a memory, the processor being configured to execute a clothing matching and keyframe extraction program stored in the memory to implement the steps of the method according to any one of claims 1 to 5 or 6.

9. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the method according to any one of claims 1 to 5 or 6.

Images