Clothing image classification method and mirror display

Abstract

This application discloses a method for classifying clothing images and a mirror display, relating to the field of smart home technology. The mirror display can determine the type of a first clothing item in an original image and add a target clothing image, obtained by synthesizing a sub-image of the first clothing item with a target background image, to a target image set. The clothing images stored in the target image set include the type of the first clothing item. Thus, the mirror display provided by embodiments of the present application can classify and store target clothing images based on the type of clothing in the target clothing image, and the mirror display is relatively versatile.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and in particular to a clothing image classification method and a mirror display. Background Art

[0002] In addition to displaying a mirror image, a mirror display (also known as a smart mirror) can also display clothing images, wherein the clothing images can be images obtained by photographing the user's clothing.

[0003] However, the functions of the mirror display in the related art are relatively simple. Summary of the Invention

[0004] This application provides a clothing image classification method and mirror display, which can solve the problem of low efficiency in clothing image classification in related technologies. The technical solution is as follows:

[0005] In one aspect, a method for classifying clothing images is provided, which is applied to a mirror display, wherein the mirror display includes a camera; the method comprises:

[0006] In response to a photo taking instruction, controlling the camera to take a photo of the clothing to obtain an original image;

[0007] determining a sub-image of a first article of clothing from the original image;

[0008] determining a type of the first clothing item;

[0009] synthesizing the first clothing sub-image with the target background image to obtain a target clothing image;

[0010] The target clothing image is added to a target image set, where the target image set is used to store clothing images of clothing of a target type, where the target type includes the type of the first clothing.

[0011] In another aspect, a mirror display is provided, comprising a processor and a camera; the processor being configured to:

[0012] In response to a photo taking instruction, controlling the camera to take a photo of the clothing to obtain an original image;

[0013] determining a sub-image of a first article of clothing from the original image;

[0014] determining a type of the first clothing item;

[0015] synthesizing the first clothing sub-image with the target background image to obtain a target clothing image;

[0016] The target clothing image is added to a target image set, where the target image set is used to store clothing images of clothing of a target type, where the target type includes the type of the first clothing.

[0017] On the other hand, a mirror display is provided, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for classifying clothing images as described in the above aspects is implemented.

[0018] On the other hand, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the method for classifying clothing images as described in the above aspects.

[0019] In yet another aspect, a computer program product comprising instructions is provided. When the computer program product is run on a computer, the computer is enabled to execute the method for classifying clothing images as described in the above aspect.

[0020] The beneficial effects of the technical solution provided by this application include at least:

[0021] This application provides a method for classifying clothing images and a mirror display. The mirror display can determine the type of a first clothing item in an original image and add a target clothing image, obtained by synthesizing a sub-image of the first clothing item with a target background image, to a target image set. The types of clothing images stored in the target image set include the type of the first clothing item. Thus, the mirror display provided by the embodiments of the present application can classify and store target clothing images based on the type of clothing in the target clothing image, and the mirror display is relatively versatile.

[0022] Furthermore, because the mirror display automatically identifies the type of the first clothing item and automatically adds the target clothing item image to the target image set, the efficiency of adding the target clothing item image to the target image set is improved. Because the target clothing item image is a composite of the first clothing item sub-image and the target background image, the background images of each clothing item in the target image set are relatively uniform, thereby ensuring a better display effect for the clothing items in the target image set. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1 is a schematic diagram of a mirror display provided in an embodiment of the present application;

[0025] Figure 2 This is a flow chart of a method for classifying clothing images provided in an embodiment of the present application;

[0026] Figure 3 is a flowchart of another clothing image classification method provided by an embodiment of the present application;

[0027] Figure 4 is a schematic diagram of an application interface of a mirror display provided in an embodiment of the present application;

[0028] Figure 5 This is a flow chart of a mirror display provided by an embodiment of the present application for cropping a target cropped sub-image from an original image;

[0029] Figure 6 This is a schematic diagram of an interface for adjusting a clothing entry method of an original image provided by an embodiment of the present application;

[0030] Figure 7 This is a flow chart of determining a sub-image of a second garment by a mirror display provided by an embodiment of the present application;

[0031] Figure 8 is a schematic diagram of an original image provided in an embodiment of the present application;

[0032] Figure 9 This is the package entry method provided in the embodiment of the present application, based on Figure 8 Schematic diagram of the target clothing image synthesized from the original image shown;

[0033] Figure 10 This is a single piece of clothing entry method provided by the embodiment of the present application, based on Figure 8 Schematic diagram of the target clothing image synthesized from the original image shown;

[0034] Figure 11 This is a single entry method for multiple clothes provided by the embodiment of the present application, based on Figure 8 Schematic diagram of the target clothing image synthesized from the original image shown;

[0035] Figure 12 This is a schematic diagram of a clothing image display interface provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0037] The embodiment of the present application provides a method for classifying clothing images, which can be applied to mirror displays. Figure 1 The mirror display includes: a camera 11 and a display panel 12. Figure 2 , the method may include:

[0038] Step 101: In response to a photo taking instruction, control a camera to take a photo of the clothing to obtain an original image.

[0039] In an embodiment of the present application, a camera may capture at least one item of clothing, thereby obtaining an image. Accordingly, the original image includes at least one sub-image of the item of clothing. The at least one item of clothing includes a first item of clothing.

[0040] Optionally, the display panel of the mirror display may display a photo control. The photo command may be triggered by a touch operation on the photo control. Alternatively, the camera of the mirror display captures a human image and can recognize a human hand gesture in the human image. The photo command may be received after the mirror display determines that the hand gesture is a target gesture.

[0041] Step 102: Determine a sub-image of a first garment from the original image.

[0042] After obtaining the original image, the mirror display can determine the number of clothing sub-images included in the original image. If this number is 1, meaning the original image includes one clothing sub-image, the mirror display can directly determine the one clothing sub-image as the first clothing sub-image. If this number is greater than 1, meaning the original image includes multiple clothing sub-images, the mirror display can determine the first clothing sub-image from the original image.

[0043] In an embodiment of the present application, if an original image includes multiple clothing sub-images, the sub-image of the first clothing item depends on the clothing entry mode of the original image. The clothing entry mode can be one of the following: single-item entry and set entry. Single-item entry can be a single item of clothing or a single item of clothing.

[0044] Therefore, after the mirror display determines that the original image includes multiple clothing sub-images, it can determine the clothing entry mode of the original image. If the mirror display determines that the clothing entry mode is for multiple clothing items to be entered individually, each of the multiple sub-images included in the original image can be used as a sub-image of the first clothing item. If the mirror display determines that the clothing entry mode is for single-item or set entry of a single clothing item, the sub-image located in the center of the original image can be used as the sub-image of the first clothing item.

[0045] Step 103: Determine the type of the first clothing item.

[0046] The type of the first clothing item may include: a parent type of the first clothing item. The parent type of the first clothing item may be one of multiple parent types (also referred to as major categories). The multiple parent types may be obtained based on the wearing position of the clothing item. The multiple parent types may include: tops and bottoms. Clothing items classified as tops refer to clothing items worn above the waist of a person. Clothing items classified as bottoms refer to clothing items worn below the waist of a person.

[0047] Optionally, multiple parent types may include, in addition to tops and bottoms, jumpsuits (also known as full-piece outfits), hats, and shoes. Jumpsuits are clothing items that are worn above and below the waist, such as dresses and jumpsuits.

[0048] Furthermore, each parent type can include multiple subtypes (also called subcategories). For example, the parent type "tops" can include multiple subtypes such as coats, sweaters, hoodies, shirts and T-shirts. The parent type "bottoms" can include multiple subtypes such as jeans, casual trousers, denim shorts and casual shorts. The parent type "dresses" can include multiple subtypes such as dresses and jumpsuits. The parent type "shoes" can include multiple subtypes such as slippers, boots and sneakers. The parent type "hats" can include multiple subtypes such as berets, peaked caps and bucket hats. Based on this, the type of the first clothing can also include: the subtype of the first clothing.

[0049] Optionally, the mirror display may crop a target cropped sub-image from the original image, where the target cropped sub-image includes a sub-image of the first clothing. Thereafter, the mirror display may identify the target cropped sub-image to obtain the type of the first clothing.

[0050] Step 104: Synthesize the first clothing sub-image with the target background image to obtain a target clothing image.

[0051] Optionally, the mirror display may superimpose the sub-image of the first clothing item on the target background image to synthesize the target clothing item image. The target background image may have a size equal to that of the original image. The pixel value of each image pixel included in the target background image may be pre-stored by the mirror display.

[0052] Step 105: Add the target clothing image to the target image set.

[0053] The target image set is used to store clothing images of clothing of a target type, where the target type may include the type of the first clothing. In this way, an electronic wardrobe can be formed so that the user can effectively manage his or her clothing.

[0054] In an embodiment of the present application, after the target clothing image is added to the target image set (i.e., the sub-image of the first clothing is entered into the target image set), if the mirror display receives a viewing instruction for the clothing image in the target image set, the clothing image in the target image set can be directly displayed.

[0055] In summary, the embodiments of the present application provide a method for classifying clothing images. A mirror display can determine the type of a first clothing item in an original image and add a target clothing image, obtained by synthesizing a sub-image of the first clothing item with a target background image, to a target image set. The types of clothing images stored in the target image set include the type of the first clothing item. Thus, the mirror display provided by the embodiments of the present application can classify and store target clothing images based on the type of clothing in the target clothing image, and the mirror display is relatively versatile.

[0056] Furthermore, because the mirror display automatically identifies the type of the first clothing item and automatically adds the target clothing item image to the target image set, the efficiency of adding the target clothing item image to the target image set is improved. Because the target clothing item image is a composite of the first clothing item sub-image and the target background image, the background images of each clothing item in the target image set are relatively uniform, thereby ensuring a better display effect for the clothing items in the target image set.

[0057] The embodiment of the present application takes the original image including multiple clothing sub-images as an example to illustrate the process of the clothing image classification method provided by the embodiment of the present application. Figure 3 , the method may include:

[0058] Step 201: In response to a photo taking instruction, control a camera to take a photo of the clothing to obtain an original image.

[0059] In an embodiment of the present application, after a user places multiple items of clothing within the field of view of a camera of a mirror display, or after the user wears multiple items of clothing and places them within the field of view of the camera, a photo-taking instruction can be triggered. Accordingly, the mirror display can respond to the photo-taking instruction and control the camera to take a photo to obtain an original image.

[0060] Optionally, the display panel of the mirror display may display a photo control. The photo command may be triggered by a user's touch operation on the photo control. Alternatively, the mirror display's camera captures a human image and can recognize a human hand gesture in the image. The photo command may be triggered by the hand gesture being a target gesture. The target gesture is a gesture used to instruct the mirror display to take a photo.

[0061] For example, a clothing management application can be installed in the mirror display. Figure 4 The user can touch the application logo 01 of the clothing management application displayed on the mirror display. In response to the user's touch operation on the application logo 01, the mirror display displays an application interface, which may include: a photo control 02.

[0062] The user can touch the photo control 02, and the mirror display can control the camera to take pictures in response to the skin photo instruction triggered by the user's touch operation on the photo control 02.

[0063] It is understandable that the display screen of the mirror display can display the mirror image while displaying the application logo of the clothing management application, and the application logo can be superimposed on the mirror image.

[0064] Step 202: crop a target cropped sub-image from the original image.

[0065] The target cropped sub-image includes a sub-image of the first clothing item. Optionally, the outline of the target cropped sub-image may be the outline of the sub-image of the first clothing item, i.e., the target cropped sub-image is the sub-image of the first clothing item. Alternatively, the sub-image of the first clothing item may be located within the target cropped sub-image, i.e., the area of ​​the target cropped sub-image is larger (e.g., slightly larger) than the area of ​​the sub-image of the first clothing item. The number of the target cropped sub-images may be one or more.

[0066] See also Figure 5 ,The process of cropping a target cropped sub-image from an original image by a mirror display may include:

[0067] Step 2021: Use the clothing detection model to process the original image to obtain multiple cropping frames.

[0068] In the embodiment of the present application, the image area where each cropping frame is located may include a sub-image of clothing. The difference between the length of each cropping frame and the length of the sub-image in the cropping frame may be equal to a target value, and the difference between the width of each cropping frame and the width of the sub-image in the cropping frame may also be equal to the target value, and the boundary line of the cropping frame does not overlap with the boundary line of the sub-image in the cropping frame.

[0069] The length of the cropping frame and the sub-image can both extend parallel to the direction of arrangement of pixels in a row of the original image, and the width of the cropping frame and the sub-image can both extend parallel to the direction of arrangement of pixels in a column of the original image. The target value can be pre-stored by the mirror display and be less than or equal to a threshold. The target value can be the sum of the lengths of four pixels in the original image.

[0070] As can be seen, the area of ​​the image region within the cropping frame is slightly larger than the area of ​​the sub-image within the cropping frame. This ensures the integrity of the clothing sub-image cropped based on the cropping frame while conserving processing resources on the mirror display. Optionally, each cropping frame can be rectangular.

[0071] It should be noted that if the original image is displayed on a display screen of a mirror display, the arrangement direction of a row of image pixels in the original image can be parallel to the extension direction of the pixel row of the display screen, and the arrangement direction of a column of image pixels in the original image can be parallel to the extension direction of the pixel column of the display screen.

[0072] The length of the clothing sub-image may refer to the length of a row of image pixels having the most image pixel columns among the multiple rows of image pixels included in the clothing sub-image. The width of the clothing sub-image may refer to the width of a column of image pixels having the most image pixel rows among the multiple columns of image pixels included in the clothing sub-image.

[0073] In an embodiment of the present application, the clothing detection model may be pre-stored in the mirror display, and the clothing detection model may be trained based on a plurality of first sample data. Each first sample data may include: a sample image, and a sub-image of the sample clothing in the sample image. Optionally, the clothing detection model may be trained using the you only look once (YOLO) algorithm. For example, the fifth version of the YOLO algorithm, namely the YOLOV5 algorithm, may be used for training. The YOLO algorithm is a unified real-time object detection algorithm.

[0074] After the mirror display captures the original image, it can input the original image into the clothing detection model. The clothing detection model can then perform clothing detection on the original image to detect multiple clothing sub-images. Furthermore, for each detected clothing sub-image, the clothing detection model can generate a cropping frame surrounding the sub-image based on the length and width of the clothing sub-image.

[0075] Step 2022: Determine a first cropping frame from multiple cropping frames.

[0076] In the embodiment of the present application, the clothing entry method of the original image can be one of the following methods: single-piece entry and set entry. Single-piece entry refers to obtaining a target clothing image based on a sub-image of a piece of clothing in the original image. Set entry refers to obtaining a target clothing image based on a set of clothing in the original image (i.e., a sub-image of a top and a sub-image of a bottom).

[0077] Furthermore, single-item entry can also be one of the following: single-item entry for a single item of clothing and single-item entry for multiple items of clothing. Single-item entry for a single item of clothing means obtaining a single target item of clothing image based on only one item of clothing sub-image in the original image. Single-item entry for multiple items of clothing means obtaining multiple target item of clothing images based on multiple item of clothing sub-images included in the original image. Each target item of clothing image is obtained based on a single item of clothing sub-image.

[0078] That is, when the clothing entry mode is set entry, the mirror display can select a sub-image of an upper garment and a sub-image of a lower garment from multiple clothing sub-images, and generate a target clothing image based on the upper garment sub-image and the lower garment sub-image. When the clothing entry mode is single-piece entry of a single garment, the mirror display can select a sub-image of a first garment from multiple clothing sub-images, and generate a target clothing image based on the sub-image of the first garment. When the clothing entry mode is single-piece entry of multiple garments, the mirror display can treat all of the multiple garments as the first garment, and then generate multiple target clothing images based on the multiple first garment sub-images. In this way, the efficiency of clothing image classification can be improved. As can be seen from the above description, when the clothing entry mode is set entry or single-piece entry of multiple garments, the mirror display can enter multiple clothing sub-images at once, which improves the efficiency of clothing sub-image entry.

[0079] In this embodiment of the present application, if the clothing entry method is to enter multiple clothing items individually, the mirror display may use each clothing item sub-image in the multiple clothing item sub-images included in the original image as a first clothing item sub-image. If the clothing entry method is to enter a set or a single clothing item, the mirror display needs to determine the first clothing item sub-image from the multiple clothing item sub-images included in the original image.

[0080] Based on this, after the mirror display obtains multiple cropping frames, it can obtain the clothing entry mode of the original image. If the mirror display determines that the clothing entry mode is: single entry of multiple clothing items, it can be determined that each of the multiple cropping frames is the first cropping frame.

[0081] If the mirror display determines that the clothing entry method is a single piece of clothing or a set of clothing, the sub-image located in the middle of the original image can be determined as the sub-image of the first clothing. Because when the user requires the mirror display to record a sub-image of a certain clothing item, the clothing item is usually placed in the center of the field of view of the mirror display's camera. Accordingly, in the original image captured by the camera, the clothing sub-image of the clothing item is usually located in the middle of the original image. Based on this, for each of the multiple cropping frames, the mirror display can determine a first distance between the center point of the cropping frame and the center point of the original image, and can determine the cropping frame with the smallest first distance among the multiple cropping frames as the first cropping frame.

[0082] In an embodiment of the present application, the mirror display can determine the position of the center point of the cropping frame in the original image based on the position of the cropping frame in the original image. The position of each center point of the cropping frame and the center point of the original image can refer to: the coordinates of the center point in the image coordinate system in which the original image is located. The position of the cropping frame in the original image can refer to: the coordinates of the outline of the cropping frame in the image coordinate system. The image coordinate system can be a coordinate system established with a vertex of the original image (for example, the upper left vertex) as the origin, the direction in which the pixel rows of the original image extend as the first coordinate axis, and the direction in which the pixel columns of the original image extend as the second coordinate axis. The first coordinate axis can be one of the horizontal axis (also called the x-axis) and the vertical axis (also called the y-axis), and the second coordinate axis can be the other of the horizontal axis and the vertical axis. For example, the first coordinate axis is the horizontal axis, and the second coordinate axis is the vertical axis.

[0083] If the cropping frame is a rectangular cropping frame, the position of the cropping frame may refer to the coordinates of two vertices of the cropping frame located on a diagonal line of the cropping frame in the image coordinate system, where the coordinates may include x-coordinates and y-coordinates. Accordingly, the x-coordinate of the center point of the cropping frame in the image coordinate system may be the average of the x-coordinates of the two vertices, and the y-coordinate of the center point of the cropping frame in the image coordinate system may be the average of the y-coordinates of the two vertices.

[0084] Step 2023: Crop the original image according to the first cropping frame to obtain a target cropped sub-image within the first cropping frame.

[0085] The mirror display can crop the original image using the boundary of the first cropping frame as a boundary line to obtain a target cropped sub-image. The position of the target cropped sub-image in the original image coincides with the position of the area where the first cropping frame is located in the original image.

[0086] It can be understood that, for a scene with multiple first cropping frames, for each first cropping frame in the multiple first cropping frames, the mirror display can crop the original image according to the first cropping frame, thereby obtaining multiple cropped sub-images.

[0087] Step 203: The mirror display performs image recognition on the target cropped sub-image to obtain the type of the first clothing.

[0088] The type of the first clothing item may include a parent type of the first clothing item. The parent type of the first clothing item may be one of multiple parent types. The multiple parent types may be classified based on the position of the clothing item being worn. The multiple parent types may include tops and bottoms. Optionally, in addition to tops and bottoms, the multiple parent types may also include dresses (also referred to as full-piece outfits), hats, and shoes.

[0089] Furthermore, each parent type can include multiple subtypes (also called subcategories). For example, the parent type "tops" can include: coats, sweaters, hoodies, shirts and T-shirts and other subtypes. The parent type "bottoms" can include: jeans, casual trousers, denim shorts and casual shorts and other subtypes. The parent type "dresses" can include: dresses and jumpsuits and other subtypes. The parent type "shoes" can include: slippers, boots and sneakers and other subtypes. The parent type "hats" can include: berets, peaked caps and bucket hats and other subtypes. Based on this, the type of the first clothing can also include: subtypes of the first clothing. In this way, the accuracy of the classification of clothing images can be ensured, and then it can be convenient for users to accurately know the clothing belonging to each subtype, thereby improving the user experience.

[0090] Since the mirror display can perform image recognition on the target cropped sub-image, which only includes the sub-image of the first clothing, it can save processing resources of the mirror display on the one hand, and ensure the accuracy of the identified type of the first clothing on the other hand.

[0091] Optionally, the mirror display may use a clothing recognition model to perform image recognition processing on the target cropped sub-image to determine the type of the first clothing item. The clothing recognition model may be pre-stored in the mirror display and may be trained based on a large amount of second sample data. Each second sample data item may include a sub-image of a sample clothing item and a type of the sample clothing item. The type includes a parent type and a child type of the sample clothing item.

[0092] Optionally, the clothing recognition model may be obtained by training a large amount of second sample data using a residual neural network (ResNet-50) algorithm. For example, the model may be obtained by training a large amount of second sample data using a ResNet algorithm with 50 network layers (i.e., ResNet-50).

[0093] In an embodiment of the present application, after obtaining the position of the first cropping frame in the image coordinate system of the original image and the type of the first clothing item, the mirror display may also record the position and the target type. For example, assuming the cropping frame is a rectangular cropping frame and the type of the first clothing item includes a parent type and a child type, the mirror display may record the position and the type of the first clothing item in the following format: [x1, y1, x2, y2, big_class_name, small_class_name].

[0094] Where (x1, y1) is the coordinate of a vertex of the first cropping frame in the image coordinate system, and (x2, y2) is the coordinate of another vertex of the first cropping frame in the image coordinate system. The vertex and the other vertex are located on a diagonal line of the first cropping frame. big_class_name is the identifier of the parent class of the first clothing item, and small_class_name is the identifier of the child class of the first clothing item.

[0095] Step 204: The mirror display detects whether the clothing entry mode is suit entry.

[0096] If the mirror display determines that the clothing entry mode of the original image is not suit entry, step 205 may be executed. If the mirror display determines that the clothing entry mode is suit entry, step 206 may be executed.

[0097] In an embodiment of the present application, the clothing entry method for the original image is pre-stored in the mirror display. A clothing management application may be installed in the mirror display. The application interface of the clothing management application may display a first option for set entry, a second option for single-piece entry, and a third option for multiple-piece entry. In response to a user selecting one of the first through third options, the mirror display may determine the clothing entry method for the original image. The default clothing entry method is single-piece entry.

[0098] For example, see Figure 4 and Figure 6 The application interface of the clothing management application installed in the mirror display may further include: clothing entry control 03. If the user needs to adjust the clothing entry method, the user may touch the clothing entry control 03. Figure 6 In response to the touch operation on the clothing entry control 03, the mirror display can display the first option 031a to the third option 033a, and the first explanation information 031b to the third explanation information 033b corresponding to the first option 031a to the third option 033a. Among them, the second option 031a is selected by default.

[0099] from Figure 6As can be seen, each of the first to third description information 031b to 033b can be text. Specifically, the first description information 031b can read: "When this option is selected, the mirror display can record two sub-images of clothing items belonging to the suit in the original image." The second description information 032b can read: "When this option is selected, the mirror display can record a sub-image of one clothing item in the original image." The third description information 033b can read: "When this option is selected, the mirror display can record sub-images of multiple clothing items in the original image."

[0100] If the user requires the mirror display to record the sub-image of the clothing in the first recording mode, the user can touch the first option 031a. In response to the user's touch operation on the first option 031a, the mirror display can determine the routine recording as the clothing recording mode of the original image and send the clothing recording mode to the mirror display.

[0101] Step 205: The mirror display combines a sub-image of the first clothing with the target background image to obtain a target clothing image.

[0102] If the mirror display determines that the clothing item entry method for the original image is single-item entry, a sub-image of the first item of clothing is superimposed on the target background image to create a target clothing image. The target background image can be the same size as the original image, and the sub-image of the first item of clothing can be superimposed in the center of the target background image. This ensures that the sub-image of the first item of clothing in the resulting target clothing image displays a better effect.

[0103] In an embodiment of the present application, if the target cropped sub-image is rectangular, the mirror display may perform image segmentation on the target cropped sub-image after obtaining the target cropped sub-image to obtain a sub-image of the first clothing item. The mirror display may then combine the sub-image of the first clothing item with the target background image to obtain the target clothing item image.

[0104] Optionally, the mirror display can use the DeeplabV3+ image segmentation algorithm to perform image segmentation processing on the target cropped sub-image, and can use a mobile network (MobileNet) algorithm (such as the MobileNetV2 algorithm) to perform feature extraction to obtain a sub-image of the first clothing. Then, the mirror display can optimize the sub-image of the first clothing. For example, the mirror display can perform interference removal processing, image denoising processing, edge smoothing processing, and image completion processing on the sub-image of the first clothing, so as to achieve the effect of optimizing the sub-image of the first clothing. Among them, the DeeplabV3+ image segmentation algorithm is an image segmentation algorithm that combines a segmentation algorithm that uses an encoder-decoder structure for image segmentation with a segmentation algorithm that uses an atrous separable convolution structure for image segmentation.

[0105] Afterwards, the mirror display may combine the optimized sub-image of the first clothing item with the target background image to obtain the target clothing image. For example, the mirror display may use the np.zeros() function to obtain a feature matrix of the target background image, and may replace the values ​​at the target positions in the feature matrix with the feature matrix of the sub-image of the first clothing item to obtain the target clothing image. Each value in the feature matrix of the target background image is 0. Each of the multiple values ​​included in the feature matrix of the sub-image of the first clothing item may be the pixel value of an image pixel in the sub-image of the first clothing item.

[0106] Since the mirror display can optimize the sub-image of the first clothing and obtain the target clothing image based on the optimized sub-image of the first clothing, it can ensure that the display effect of the obtained target clothing image is high.

[0107] Since the DeeplabV3+ image segmentation algorithm is highly time-sensitive, occupies fewer processing resources, and is highly robust, the DeeplabV3+ image segmentation algorithm can be used to improve the extraction efficiency of the sub-image of the first clothing, and ensure that the segmented sub-image of the first clothing is not affected by lighting and ambient color, thereby ensuring that the display effect of the sub-image of the first clothing is better.

[0108] Step 206: The mirror display determines a sub-image of a second piece of clothing from the plurality of sub-images of clothing according to the type of the first piece of clothing.

[0109] If the mirror display determines that the clothing label entry mode of the original image is set, a second clothing sub-image can be determined from the multiple clothing sub-images. The second clothing item and the first clothing item are of the same type: tops and bottoms, respectively. For example, if the first clothing item is of the same type: tops, then the second clothing item is of the same type: bottoms. If the first clothing item is of the same type: bottoms, then the second clothing item is of the same type: tops.

[0110] In the embodiment of the present application, after the mirror display determines that the clothing entry mode of the original image is suit entry, it can detect whether the number of sub-images of the multiple clothing items included in the original image (i.e., the number of multiple cropping frames included in the original image) is greater than 2. If the mirror display determines that the number of sub-images of the multiple clothing items is equal to 2, the sub-images other than the sub-image of the first clothing item can be directly determined as the sub-image of the second clothing item.

[0111] If the mirror display determines that the number of the plurality of clothing sub-images is greater than 2 (ie at least three), then refer to Figure 7 The process of the mirror display determining the sub-image of the second clothing may include:

[0112] Step 2061: Determine the types of at least two candidate clothes other than the first clothes among the plurality of clothes.

[0113] In the embodiment of the present application, for each second cropping frame other than the first cropping frame among the plurality of cropping frames, the mirror display can crop the original image according to the second cropping frame to obtain an alternative cropped sub-image located within the second cropping frame. Each alternative cropped sub-image can include a sub-image of an alternative garment.

[0114] Afterwards, the mirror display may perform image recognition on each candidate cropped sub-image to obtain a type of candidate clothing, thereby obtaining at least two types of candidate clothing in addition to the first clothing.

[0115] The implementation process of the mirror display cropping the original image according to the second cropping frame can be referred to the implementation process of step 2023. The implementation process of the mirror display performing image recognition on the candidate cropped sub-image in the second cropping frame to obtain the type of candidate clothing can be referred to the implementation process of step 203, and this embodiment of the application will not be repeated here.

[0116] Step 2062: Based on the types of the at least two candidate clothes and the type of the first clothes, determine a sub-image of the second clothes from the sub-images of the at least two candidate clothes.

[0117] In an embodiment of the present application, if there is an initial clothing item among at least two alternative clothing items, and the type of the initial clothing item and the type of the first clothing item belong to tops and bottoms respectively, the mirror display can directly determine the sub-image of the initial clothing item as the sub-image of the second clothing item.

[0118] If there are multiple initial clothes among the at least two candidate clothes, the process of the mirror display determining the sub-image of the second clothes from the sub-images of the at least two candidate clothes may include:

[0119] The mirror display determines a plurality of initial clothing sub-images from the at least two candidate clothing sub-images based on the types of the at least two second clothing items and the type of the first clothing item. Then, for each initial clothing sub-image, the mirror display may determine a second distance in a target direction between a center point of a second cropping frame including the initial clothing sub-image and a center point of the first cropping frame, and determine the initial clothing sub-image within the second cropping frame where the second distance is less than a distance threshold as the second clothing sub-image.

[0120] The target direction is parallel to the arrangement direction of a row of image pixels in the original image. The distance threshold can be pre-stored in the mirror display. For example, the distance threshold can be a length of 10 pixels in the original image. Because the mirror display can determine that a sub-image in a second cropping frame that is of a different type from the first clothing item and whose second distance between its center point and the center point of the first cropping frame is less than the distance threshold is a sub-image of the second clothing item, the accuracy of the determined sub-image of the second clothing item can be ensured to be high even when multiple sub-images of other clothing items are present near the sub-image of the first clothing item. Each of the other clothing items is of a different type from the first clothing item.

[0121] Optionally, after determining that the clothing entry mode is set to suit entry, the mirror display may further detect whether the first clothing item is a dress, a hat, or shoes. If the mirror display determines that the first clothing item is a dress, a hat, or shoes, step 205 may be executed. If the mirror display determines that the clothing item in the sub-image of the first clothing item is not a dress, a hat, or a shoe, a second clothing item sub-image may be determined from the plurality of clothing item sub-images.

[0122] Step 207 : Synthesize the first clothing sub-image, the second clothing sub-image, and the target background image to obtain a target clothing image.

[0123] The relative positions of the first clothing sub-image and the second clothing sub-image in the target clothing image may be the same as those in the original image.

[0124] It can be understood that before obtaining the target clothing image, the mirror display can perform image segmentation processing on the target cropped sub-image including the sub-image of the first clothing and the alternative cropped sub-image including the sub-image of the second clothing, so as to obtain the sub-image of the first clothing and the sub-image of the second clothing.

[0125] Among them, the mirror display can perform image segmentation processing on the cropped sub-image. Please refer to the relevant implementation process in step 205, and the embodiment of the present application will not be repeated here.

[0126] For example, assume that the original image captured by the mirror display is Figures 8 to 11 The image A shown is from Figure 8 It can be seen that image A includes two sub-images of clothing: sub-image a1 and sub-image a2.

[0127] Assuming that the clothing entry method is set entry, please refer to Figure 9 , the mirror display can superimpose sub-image a1 and sub-image a2 on the target background image to synthesize Figure 9 The target clothing image shown is Image B.

[0128] Assuming that the clothing entry mode is single piece entry of a single piece of clothing, since the center point of the cropping frame surrounding the sub-image a1 is closest to the center point of the image A, the mirror display can superimpose the sub-image a1 on the target background image to synthesize Figure 10 The target clothing image shown is Image C.

[0129] Assuming that the clothing entry method is to enter multiple pieces of clothing individually, the mirror display can superimpose the sub-image a1 on the target background image to synthesize Figure 11 The target clothing image shown is image C, and the sub-image a2 can be superimposed on the target background image to synthesize Figure 11 The target clothing image shown is Image D.

[0130] Step 208: The mirror display adds the target clothing image to the target image set.

[0131] The target image set is used to store clothing images of clothing of a target type, where the target type includes a first clothing type.

[0132] It is understood that if the clothing entry method is single-piece entry, the target image set is the image set corresponding to the first clothing type. The clothing type in each clothing image stored in the image set corresponding to the first clothing type is the same as the first clothing type. For example, the clothing subtype in each clothing image is the same as the subtype of the first clothing. If the clothing entry method is set entry, each clothing image stored in the target image set is an image of a set (i.e., a top and a bottom).

[0133] In an embodiment of the present application, after the mirror display adds the target clothing image to the target image set, if the mirror display receives a viewing instruction for the clothing image in the target image set, the clothing image in the target image set can be directly displayed.

[0134] For example, see Figure 12 The application interface of the clothing management application installed in the mirror display may further include: My Wardrobe Control 04. The mirror display may display a plurality of image collection selection controls in response to a user's touch operation on My Wardrobe Control 04. The selection control of each image collection may display the identification of the image collection (e.g., the type of clothing in the clothing images included in the image collection). For example, Figure 12 Four selection controls for image sets are shown: a first selection control 041 to a fourth selection control 044. The first selection control 041 is in a selected state by default.

[0135] It is understood that if the first selection control 041 is in a selected state, the mirror display can display all recorded clothing images. If the second selection control 041 is in a selected state, the mirror display can display clothing images of the type of tops. If the third selection control 041 is in a selected state, the mirror display can display clothing images of the type of bottoms. If the fourth selection control 041 is in a selected state, the mirror display can display clothing images of suits.

[0136] If the user needs to view clothing images of a certain type of clothing, the user can touch the selection control of the image collection storing clothing images of the type of clothing. Figure 12 If the user wishes to view an image of a garment of the upper garment type, they may touch the first selection control 041. Accordingly, the mirror display may, in response to the first selection control 041, send a request to obtain an image of the upper garment to the mirror display. In response to the request, the mirror display may send an image of the upper garment to the mirror display. The mirror display may then display the image of the upper garment for the user to view.

[0137] It's understandable that when a user is recording an image of clothing through a mirror display, other users may also be within the field of view of the mirror display's camera. This can result in the presence of multiple sub-images of clothing worn on different people in the original image captured by the mirror display. In this case, after receiving the original image, the mirror display can first perform body recognition to obtain multiple body cropping frames. Each body cropping frame contains a body sub-image.

[0138] Since the user is typically located in the center of the camera's field of view when recording an image of clothing, the mirror display can determine, among the multiple human cropping frames, the human cropping frame whose center point is closest to the center point of the original image as the target human cropping frame. The mirror display can then process the human sub-image in the target human cropping frame using steps 201 to 208 described above to obtain the target clothing image. This shows that the mirror display provided by the embodiment of the present application can accurately record clothing sub-images even in scenarios where multiple human sub-images exist in the original image.

[0139] It should be noted that the order of the steps in the clothing image classification method provided in the embodiments of this application can be adjusted appropriately, and the number of steps can be increased or decreased depending on the circumstances. For example, step 201 can be deleted as needed; or steps 204, 206, 207, and 208 can be deleted as needed. Any variations that can be readily conceived by a person skilled in the art within the scope of the technology disclosed in this application are intended to be covered by the scope of protection of this application, and therefore will not be described in detail.

[0140] In summary, the embodiments of the present application provide a method for classifying clothing images. A mirror display can determine the type of a first clothing item in an original image and add a target clothing image, obtained by synthesizing a sub-image of the first clothing item with a target background image, to a target image set. The types of clothing images stored in the target image set include the type of the first clothing item. Thus, the mirror display provided by the embodiments of the present application can classify and store target clothing images based on the type of clothing in the target clothing image, and the mirror display is relatively versatile.

[0141] Furthermore, because the mirror display automatically identifies the type of the first clothing item and automatically adds the target clothing item image to the target image set, the efficiency of adding the target clothing item image to the target image set is improved. Because the target clothing item image is a composite of the first clothing item sub-image and the target background image, the background images of each clothing item in the target image set are relatively uniform, thereby ensuring a better display effect for the clothing items in the target image set.

[0142] The present application provides a mirror display that can be used to implement the clothing image classification method provided in the above method embodiment. The mirror display includes a processor and a camera. The processor is used to:

[0143] In response to a photo taking instruction, controlling the camera to take a photo of the clothing to obtain an original image;

[0144] determining a sub-image of a first article of clothing from the original image;

[0145] determining a type of the first garment;

[0146] synthesizing the first clothing sub-image with the target background image to obtain a target clothing image;

[0147] The target clothing image is added to a target image set, where the target image set is used to store clothing images of clothing of a target type, where the target type includes the type of the first clothing.

[0148] Optionally, the processor is used to:

[0149] cropping a target cropped sub-image from the original image, where the target cropped sub-image includes a sub-image of the first clothing;

[0150] Image recognition is performed on the target cropped sub-image to obtain the type of the first clothing.

[0151] Optionally, the original image includes: a plurality of clothing sub-images. The processor is configured to:

[0152] The clothing detection model is used to process the original image to obtain multiple cropping frames, where the image area where each cropping frame is located includes a sub-image of the clothing;

[0153] For each cropping frame in the plurality of cropping frames, determining a first distance between a center point of the cropping frame and a center point of the original image;

[0154] Determine a cropping frame with the smallest first distance among the multiple cropping frames as a first cropping frame;

[0155] The original image is cropped according to the first cropping frame to obtain a target cropped sub-image located within the first cropping frame.

[0156] Optionally, the target cropped sub-image is rectangular. The processor is further configured to:

[0157] Perform image segmentation processing on the target cropped sub-image to obtain a sub-image of the first clothing.

[0158] Optionally, if the clothing entry mode of the original image is single piece entry, the target image set is an image set corresponding to the type.

[0159] Optionally, the original image includes: a plurality of sub-images of clothing, wherein the plurality of clothing includes the first clothing. The processor is further configured to:

[0160] If the clothing entry mode of the original image is set entry, then according to the type of the first clothing, a second clothing sub-image is determined from the plurality of clothing sub-images, where the second clothing type and the first clothing type belong to tops and bottoms respectively;

[0161] The first clothing sub-image, the second clothing sub-image, and the target background image are synthesized to obtain a target clothing image.

[0162] Optionally, the processor is used to:

[0163] determining types of at least two candidate clothes other than the first clothes among the plurality of clothes;

[0164] Based on the types of the at least two candidate clothes and the type of the first clothes, a sub-image of the second clothes is determined from the sub-images of the at least two candidate clothes.

[0165] Optionally, there are multiple initial clothes in the at least two candidate clothes, and the type of each initial clothes and the type of the first clothes are tops and bottoms respectively. The processor is used to:

[0166] determining a plurality of initial clothing sub-images from the at least two candidate clothing sub-images based on the at least two second clothing types and the first clothing type;

[0167] For each initial clothing sub-image, determining a second distance between a center point of a second cropping frame including the initial clothing sub-image and a center point of a first cropping frame including the first clothing sub-image in a target direction, where the target direction is parallel to an arrangement direction of a row of image pixels in the original image;

[0168] The initial clothing sub-image within the second cropping frame, where the second distance is less than the distance threshold, is determined as the second clothing sub-image.

[0169] In summary, the embodiments of the present application provide a mirror display that can determine the type of a first item of clothing in an original image and add a target clothing image, obtained by synthesizing a sub-image of the first item of clothing with a target background image, to a target image set. The types of clothing images stored in the target image set include the type of the first item of clothing. Thus, the mirror display provided by the embodiments of the present application can classify and store target clothing images based on the type of clothing in the target clothing image, and the mirror display is relatively versatile.

[0170] Furthermore, because the mirror display automatically identifies the type of the first clothing item and automatically adds the target clothing item image to the target image set, the efficiency of adding the target clothing item image to the target image set is improved. Because the target clothing item image is a composite of the first clothing item sub-image and the target background image, the background images of each clothing item in the target image set are relatively uniform, thereby ensuring a better display effect for the clothing items in the target image set.

[0171] The embodiment of the present application provides a mirror display, which may include a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the clothing image classification method provided in the above embodiment is implemented, for example Figure 2 or Figure 3 The method shown.

[0172] The embodiment of the present application provides a computer-readable storage medium in which a computer program is stored. The computer program is loaded by a processor and executes the clothing image classification method provided in the above embodiment, for example Figure 2 or Figure 3 The method shown.

[0173] The present application also provides a computer program product including instructions. When the computer program product is run on a computer, the computer is caused to execute the clothing image classification method provided by the above method embodiment. For example, Figure 2 or Figure 3 The method shown.

[0174] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0175] It should be understood that the term "and / or" used herein indicates that three relationships can exist. For example, "A and / or B" can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. Furthermore, the term "at least one" in this application means one or more, and the term "plurality" in this application means two or more.

[0176] In this application, the terms "first," "second," and the like are used to distinguish between identical or similar items with substantially the same function or effect. It should be understood that "first," "second," and "nth" do not have a logical or temporal dependency, nor do they limit the quantity or order of execution. For example, without departing from the scope of the various examples described, a first item of clothing may be referred to as a second item of clothing, and similarly, a second item of clothing may be referred to as a first item of clothing.

[0177] The above description is merely an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for classifying clothing images, characterized in that: Applied to a mirror display, the mirror display includes a camera; the method includes: In response to a photographing instruction, controlling the camera to photograph the clothing to obtain an original image; determining a sub-image of a first article of clothing from the original image; determining a type of the first clothing item; If the original image includes multiple clothing sub-images, and the multiple clothing items include the first clothing item, determining a clothing entry mode for the original image, wherein the clothing entry mode includes single-piece entry or set entry. The single-piece entry includes single-piece entry of a single clothing item and single-piece entry of multiple clothing items. The single-piece entry of a single clothing item refers to obtaining a target clothing image based on a sub-image of a clothing item in the original image; the single-piece entry of multiple clothing items refers to obtaining multiple target clothing images based on multiple clothing sub-images included in the original image; each target clothing image is obtained based on a sub-image of a clothing item. If the clothing entry mode is the single piece entry, synthesizing a sub-image of the first clothing with a target background image to obtain a target clothing image; If the clothing entry mode is the suit entry, the types of at least two candidate clothing items other than the first clothing item among the multiple clothing items are determined, and there are multiple initial clothing items among the at least two candidate clothing items, and the type of each initial clothing item and the type of the first clothing item belong to tops and bottoms respectively; based on the types of the at least two candidate clothing items and the type of the first clothing item, multiple sub-images of the initial clothing items are determined from the sub-images of the at least two candidate clothing items; for each sub-image of the initial clothing item, a second distance between a center point of a second cropping frame including the sub-image of the initial clothing item and a center point of a first cropping frame including the sub-image of the first clothing item is determined in a target direction, where the target direction is parallel to an arrangement direction of a row of image pixels in the original image; the sub-image of the initial clothing item within the second cropping frame whose second distance is less than a distance threshold is determined as the sub-image of the second clothing item; the sub-image of the first clothing item and the sub-image of the second clothing item are synthesized with a target background image to obtain a target clothing image; Adding the target clothing image to a target image set, the target image set being used to store clothing images of a target type of clothing, where the target type includes the type of the first clothing; wherein, if the clothing entry mode is the single-piece entry, the target image set is an image set corresponding to the first clothing type; and if the clothing entry mode is the set entry, each clothing image stored in the target image set is an image of a set of clothing; In response to a viewing instruction for a clothing image in the target image set, the clothing image in the target image set is displayed.

2. The method according to claim 1, characterized in that The determining of the first clothing sub-image from the original image includes: cropping a target cropped sub-image from the original image, wherein the target cropped sub-image includes a sub-image of the first clothing; The determining the type of the first clothing comprises: Image recognition is performed on the target cropped sub-image to obtain the type of the first clothing.

3. The method according to claim 2, characterized in that The step of cropping a target cropped sub-image from the original image includes: Processing the original image using a clothing detection model to obtain a plurality of cropping frames, wherein an image region where each cropping frame is located includes a sub-image of clothing; For each of the plurality of cropping frames, determining a first distance between a center point of the cropping frame and a center point of the original image; determining a cropping frame with the smallest first distance among the plurality of cropping frames as a first cropping frame; The original image is cropped according to the first cropping frame to obtain a target cropped sub-image located within the first cropping frame.

4. The method according to claim 3, characterized in that The target cropped sub-image is rectangular; before synthesizing the first clothing sub-image with the target background image to obtain a target clothing image, the method further includes: Perform image segmentation processing on the target cropped sub-image to obtain a sub-image of the first clothing.

5. A mirror display, characterized in that: The mirror display includes a processor and a camera; the processor is configured to: In response to a photographing instruction, controlling the camera to photograph the clothing to obtain an original image; determining a sub-image of a first article of clothing from the original image; determining a type of the first clothing item; If the original image includes multiple clothing sub-images, and the multiple clothing items include the first clothing item, determining a clothing entry mode for the original image, wherein the clothing entry mode includes single-piece entry or set entry. The single-piece entry includes single-piece entry of a single clothing item and single-piece entry of multiple clothing items. The single-piece entry of a single clothing item refers to obtaining a target clothing image based on a sub-image of a clothing item in the original image; the single-piece entry of multiple clothing items refers to obtaining multiple target clothing images based on multiple clothing sub-images included in the original image; each target clothing image is obtained based on a sub-image of a clothing item. If the clothing entry mode is the single piece entry, synthesizing a sub-image of the first clothing with a target background image to obtain a target clothing image; If the clothing entry mode is the suit entry, determining the types of at least two candidate clothing items other than the first clothing item among the plurality of clothing items, wherein there are multiple initial clothing items among the at least two candidate clothing items, and the type of each of the initial clothing items and the type of the first clothing item are tops and bottoms, respectively; Based on the types of the at least two candidate clothing items and the type of the first clothing item, a plurality of initial clothing item sub-images are determined from the at least two candidate clothing item sub-images; for each of the initial clothing item sub-images, a second distance in a target direction is determined between a center point of a second cropping frame including the initial clothing item sub-image and a center point of a first cropping frame including the first clothing item sub-image, the target direction being parallel to an arrangement direction of a row of image pixels in the original image; the initial clothing item sub-image within the second cropping frame where the second distance is less than a distance threshold is determined as a second clothing item sub-image; and the first clothing item sub-image and the second clothing item sub-image are synthesized with a target background image to obtain a target clothing item image; Adding the target clothing image to a target image set, the target image set being used to store clothing images of a target type of clothing, where the target type includes the type of the first clothing; wherein, if the clothing entry mode is the single-piece entry, the target image set is an image set corresponding to the first clothing type; and if the clothing entry mode is the set entry, each clothing image stored in the target image set is an image of a set of clothing; In response to a viewing instruction for a clothing image in the target image set, the clothing image in the target image set is displayed.

6. The mirror display according to claim 5, characterized in that The processor is configured to: cropping a target cropped sub-image from the original image, wherein the target cropped sub-image includes a sub-image of the first clothing; Image recognition is performed on the target cropped sub-image to obtain the type of the first clothing.

Images