Product recommendation and image detection method, device, equipment and storage medium

Abstract

The embodiment of the present invention provides a product recommendation and image detection method, device, equipment and storage medium, the method comprising: obtaining a face image; extracting a target skin area image from the face image; identifying feature information of a target object in the target skin area image. Thus, accurate recognition of feature information related to the target object in the target skin area is achieved, so that targeted product recommendation and other services can be provided to users based on the feature information.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a product recommendation and image detection method, device, equipment and storage medium. Background Art

[0002] With the development of Internet technology, people can buy all kinds of items they need through the Internet without leaving home. For example, they can buy all kinds of goods online through various shopping applications (APPs).

[0003] In the online shopping scenario, targeted product recommendations can help users find the products they need more quickly.

[0004] At present, a commonly used product recommendation method is to search for a corresponding product list based on the product category information selected by the user on the product search interface or some keyword information entered, and recommend the product list to the user so that the user can further select the desired product from it. This method is not user-friendly and the recommendation accuracy is not high. Summary of the invention

[0005] Embodiments of the present invention provide a product recommendation and image detection method, apparatus, device and storage medium, which can accurately detect the characteristics of preset target objects (such as wrinkles) in a target area of ​​a user's body (such as a facial skin area) so as to perform subsequent processing such as product recommendation based on the characteristics.

[0006] In a first aspect, an embodiment of the present invention provides an image detection method, the method comprising:

[0007] Get face image;

[0008] Extracting a target skin area image from the face image;

[0009] Identify feature information of a target object in the target skin area image.

[0010] In a second aspect, an embodiment of the present invention provides an image detection device, the device comprising:

[0011] An acquisition module, used for acquiring a face image;

[0012] An extraction module, used to extract a target skin area image from the face image;

[0013] The recognition module is used to recognize feature information of the target object in the target skin area image.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the image detection method described in the first aspect.

[0015] In a fourth aspect, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the image detection method described in the first aspect.

[0016] In a fifth aspect, an embodiment of the present invention provides a product recommendation method, the method comprising:

[0017] Collecting facial images according to user operations;

[0018] Extracting a target skin area image from the face image, and identifying feature information of a target object in the target skin area image;

[0019] Recommended products are obtained according to the characteristic information of the target object, and the recommended products are displayed on a terminal interface.

[0020] In a sixth aspect, an embodiment of the present invention provides a product recommendation device, the device comprising:

[0021] An image acquisition module, used to acquire facial images according to user operations;

[0022] An image processing module, used to extract a target skin area image from the face image and identify feature information of a target object in the target skin area image;

[0023] The product recommendation module is used to obtain recommended products based on the characteristic information of the target object and display the recommended products on the terminal interface.

[0024] In the seventh aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the product recommendation method as described in the fifth aspect.

[0025] In an eighth aspect, an embodiment of the present invention provides a non-temporary machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor can at least implement the product recommendation method as described in the fifth aspect.

[0026] In a ninth aspect, an embodiment of the present invention provides a product recommendation method, the method comprising:

[0027] Collect human body images according to user operations;

[0028] Extracting a target part image from the human body image, and identifying feature information of a target object in the target part image;

[0029] Recommended products are obtained according to the characteristic information of the target object, and the recommended products are displayed on a terminal interface.

[0030] In a tenth aspect, an embodiment of the present invention provides a product recommendation device, the device comprising:

[0031] An image acquisition module, used to acquire human body images according to user operations;

[0032] An image processing module, used to extract a target part image from the human body image and identify feature information of a target object in the target part image;

[0033] The product recommendation module is used to obtain recommended products based on the characteristic information of the target object and display the recommended products on the terminal interface.

[0034] In the eleventh aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein an executable code is stored on the memory, and when the executable code is executed by the processor, the processor can at least implement the product recommendation method as described in the ninth aspect.

[0035] In the twelfth aspect, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the product recommendation method described in the ninth aspect.

[0036] In a thirteenth aspect, an embodiment of the present invention provides a beauty makeup method, the method comprising:

[0037] Collecting facial images;

[0038] Extracting a target skin area image from the face image, and identifying feature information of a target object in the target skin area image;

[0039] Display the beauty effect corresponding to the characteristic information of the target object.

[0040] In a fourteenth aspect, an embodiment of the present invention provides a beauty makeup device, the device comprising:

[0041] An image acquisition module, used for acquiring facial images;

[0042] An image processing module, used to extract a target skin area image from the face image and identify feature information of a target object in the target skin area image;

[0043] The beauty display module is used to display the beauty effects corresponding to the feature information of the target object.

[0044] In the fifteenth aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the beauty method described in the thirteenth aspect.

[0045] In the sixteenth aspect, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the beauty method described in the thirteenth aspect.

[0046] In an embodiment of the present invention, taking the recommendation of anti-wrinkle products to users as an example, in order to be able to recommend suitable anti-wrinkle products to users in a targeted manner according to the wrinkles in the user's forehead area, for example, the following method can be used: obtaining a face image of the user, extracting a target skin area image, such as an image of the forehead skin area, from the face image, and then identifying the characteristic information of the target object in the target skin area image. The target object is, for example, wrinkles, and its characteristic information is, for example, level information reflecting the severity of the wrinkles. Finally, according to the characteristic information of the target object in the identified target skin area, corresponding product recommendations are made, so that the product recommended to the user matches the characteristic information of the target object in its target skin area. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 A flowchart of an image detection method provided by an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of a face image provided by an embodiment of the present invention;

[0050] Figure 3 A flowchart of a method for extracting a target skin area image provided by an embodiment of the present invention;

[0051] Figure 4 A schematic diagram of the principle of a method for extracting a target skin area image provided by an embodiment of the present invention;

[0052] Figure 5 A flowchart of a method for determining a pixel position of a target object in a target skin area image provided by an embodiment of the present invention;

[0053] Figure 6 A schematic diagram of the composition of a second model provided by an embodiment of the present invention;

[0054] Figure 7 A schematic diagram of a prediction area setting effect provided by an embodiment of the present invention;

[0055] Figure 8 A schematic diagram of another second model provided by an embodiment of the present invention;

[0056] Fig. 9 A flowchart of a commodity recommendation method provided by an embodiment of the present invention;

[0057] Fig.10 A schematic diagram of an anti-wrinkle product recommendation scenario provided by an embodiment of the present invention;

[0058] Fig.11 A flowchart of a commodity recommendation method provided by an embodiment of the present invention;

[0059] Fig.12 A schematic diagram of the structure of an image detection device provided by an embodiment of the present invention;

[0060] Fig.13 For Fig.12 A schematic diagram of the structure of an electronic device corresponding to the image detection device provided in the illustrated embodiment;

[0061] Fig.14 A schematic diagram of the structure of a commodity recommendation device provided by an embodiment of the present invention;

[0062] Fig.15 For Fig.14 A schematic diagram of the structure of an electronic device corresponding to the commodity recommendation device provided in the illustrated embodiment;

[0063] Fig.16 A schematic diagram of the structure of a commodity recommendation device provided by an embodiment of the present invention;

[0064] Fig.17 For Fig.16 A schematic diagram of the structure of an electronic device corresponding to the commodity recommendation device provided in the illustrated embodiment;

[0065] Fig.18 A flowchart of a beauty makeup method provided by an embodiment of the present invention;

[0066] Fig.19 A schematic diagram of the structure of a beauty makeup device provided by an embodiment of the present invention;

[0067] Fig. 20 For Fig.19 A schematic diagram of the structure of an electronic device corresponding to the beauty makeup device provided in the illustrated embodiment. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] The terms used in the embodiments of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention.

[0070] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0071] In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0072] The image detection method provided in the embodiment of the present invention can be performed by an electronic device, which can be a terminal device such as a PC, a laptop, a smart phone, or a server. The server can be a physical server including an independent host, or a virtual server, or a cloud server.

[0073] The image detection method provided by the embodiment of the present invention can be applied to application scenarios such as online shopping. Taking the online shopping application scenario as an example, at this time, targeted product recommendations can be made for the corresponding user based on the feature information detected from the input face image based on the image detection method.

[0074] Figure 1 A flowchart of an image detection method provided by an embodiment of the present invention is as follows: Figure 1 As shown, the image detection method may include the following steps:

[0075] 101. Obtain a face image.

[0076] 102. Extract the target skin area image from the face image.

[0077] 103. Identify feature information of a target object in the target skin area image.

[0078] In this embodiment, the target skin area image refers to the image area in the face image corresponding to the target skin area, the target object refers to a certain object contained in the target skin area, and the characteristic information of the target object generally refers to some characteristic information that can reflect the visual characteristics of the target object.

[0079] In practical applications, different target skin regions can be selected according to different detection tasks. For example, if the current detection task is to detect wrinkles (forehead wrinkles) in the forehead region of a face image, then the target skin region image refers to the forehead skin region in the face image, the target object is the wrinkles in the forehead skin region, and the characteristic information of the wrinkles can be characteristic information reflecting the specific location and severity of the wrinkles, such as pixel location and level information.

[0080] Among them, for the target object of wrinkles, multiple levels of information can be pre-defined, for example, including the following four levels of information: no wrinkles, mild wrinkles, moderate wrinkles, and severe wrinkles.

[0081] For another example, if the current detection task is to detect wrinkles in the eye corners of a face image, i.e., crow's feet, then the target skin region image refers to the eye corner skin region in the face image, and the target object is the wrinkles in the eye corner skin region. Of course, for the wrinkle detection task, the target skin region may include other parts such as the neck, stomach, etc. in addition to the facial skin region.

[0082] For another example, if the current detection task is to detect pimples in the cheek area of ​​a face image, then at this time, the target skin area image refers to the cheek skin area in the face image, and the target object is the pimples in the cheek skin area. The characteristic information of the pimples may be size, number and other information.

[0083] In practical applications, based on the detection results of the characteristic information of the target object in the target skin area image, matching products can be recommended to the user, and can also be used as a basis for providing other services, but the present invention is not limited thereto.

[0084] In practical applications, when the image detection task requires processing the user's face image, the user's face image can be obtained based on the user's operation, and the obtained face image completely includes the image of the user's entire head area, such as Figure 2 as shown in .

[0085] The user operation refers to that the user can operate the terminal device to take a face image in real time, or the user can select a face image from recently taken faces.

[0086] After obtaining the user's face image, it is necessary to extract the target skin area image from the face image, and then identify the feature information of the target object in the target skin area image.

[0087] Combine the following Figure 3 and Figure 4 The illustrated embodiment introduces a method for implementing the above step 102 .

[0088] Figure 3 A flowchart of a method for extracting a target skin area image provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the method may include the following steps:

[0089] 301. Generate a first mask image corresponding to the face image, wherein, in the first mask image, pixels corresponding to the skin area have a first grayscale value, and pixels corresponding to the non-skin area have a second grayscale value.

[0090] 302. Determine a plurality of facial key points corresponding to the target skin area.

[0091] 303. Determine a position range corresponding to the target skin area in the first mask image according to the multiple facial key points.

[0092] 304. Extract a target skin area image corresponding to the position range from the face image.

[0093] For ease of understanding, combined Figure 4 The target skin region image extraction process provided in this embodiment is exemplarily described. In this embodiment, it is assumed that the target skin region is the forehead skin region.

[0094] like Figure 4 As shown in , it is assumed that after obtaining the face image 401 illustrated in the figure, on the one hand, a first mask image 402 corresponding thereto is generated. The first mask image 402 can be implemented by using existing related technologies, which will not be described in detail here. The first mask image 402 has the same size as the face image 401. From the final result, in short, in the first mask image 402, the pixels corresponding to the skin area in the face image 401 are set to the first gray value (for example, the gray value 255 corresponding to white), and the pixels corresponding to the non-skin area (such as eyebrows, hair and other background areas) in the face image 401 are set to the second gray value (for example, the gray value 0 corresponding to black), thereby realizing the detection of the skin area of ​​interest.

[0095] Based on the generation of the first mask image 402 , the purpose of identifying the skin area included in the face image 401 can be achieved.

[0096] On the other hand, face detection processing is performed on the face image 401 to obtain a number of face key points contained in the face, including a plurality of face key points corresponding to the forehead skin area as the target skin area.

[0097] Specifically, a model capable of realizing face detection function can be pre-trained, such as a convolutional neural network (CNN) model or other machine learning model, and face detection is performed through the model to obtain a number of facial key points. These facial key points are used to locate the positions of key areas of the face. Figure 4 As shown in , it is assumed that face detection is performed on face image 401 to obtain face image 403, wherein the detected face key points may include pixel positions for locating key areas such as eyebrows, eyes, nose, mouth, and facial contours.

[0098] Since the target skin area is the forehead skin area, multiple facial key points corresponding to the forehead skin area can be determined from the multiple facial key points contained in the face image 403. The multiple facial key points refer to facial key points that can assist in locating the forehead skin area, such as multiple facial key points corresponding to eyebrows.

[0099] Finally, based on the multiple facial key points corresponding to the eyebrows contained in the face image 403, the position range corresponding to the forehead skin area can be determined in the first mask image 402, such as Figure 4 Specifically, a curve can be fitted based on the pixel positions of multiple facial key points corresponding to the eyebrows, the pixel position of the curve is located in the first mask image 402, and the skin area (i.e., the white area) above the curve is determined to be the position range corresponding to the forehead skin area.

[0100] Afterwards, the position range is located in the face image 401, and the image area falling within the position range is cut out, which is the forehead skin area image.

[0101] In summary, through the above process, the target skin area image (such as the forehead skin area image) can be extracted from the face image.

[0102] After obtaining the target skin region image (such as the forehead skin region image), feature information of the target object (such as wrinkles) contained therein may be identified.

[0103] As mentioned above, the characteristic information of the target object may include pixel position and / or level information. The following describes the recognition process of these two types of characteristic information respectively.

[0104] Optionally, when the characteristic information of the target object includes the level information of the target object, the target skin area image may be input into the first model, so as to output the level information of the target object in the target skin area image through the first model.

[0105] From a functional perspective, the first model is a classification model that can achieve classification and recognition of the level of the target object. As mentioned above, multiple levels can be preset, such as the following levels: none (meaning no target object, such as wrinkles), mild, moderate, and severe.

[0106] The first model can be implemented as a variety of neural network models, such as CNN.

[0107] The following is a brief introduction to the training process of the first model.

[0108] First, a plurality of training sample images are obtained, wherein each training sample image is an image containing a target skin area. Taking the target skin area as the forehead skin area as an example, a plurality of face images containing the entire face area can be collected, and the forehead area image can be extracted from these face images as the training sample image.

[0109] Secondly, the labeling information corresponding to each of the plurality of training sample images is determined, and the labeling information includes the level information of the target object in the training sample image. Afterwards, the first model is trained until it converges according to the plurality of training sample images and the labeling information corresponding to each of the plurality of training sample images.

[0110] In practical applications, for example, if the target object is wrinkles, annotation reference information can be pre-set, and the annotation reference information refers to the correspondence between wrinkle images and wrinkle levels. Through the annotation reference information, annotators can intuitively know what level of wrinkles should be annotated. Based on the annotation reference information, annotators can annotate wrinkle levels according to the conditions of wrinkles in each training sample image.

[0111] Optionally, when the characteristic information of the target object includes the pixel position of the target object (i.e., the pixel position of the target object in the target skin area image), the target skin area image can be input into the second model to output the pixel position of the target object in the target skin area image through the second model.

[0112] It can be seen that a second model can be pre-trained to realize the recognition of the pixel position of the target object in the target skin area image.

[0113] Among them, from a functional perspective, the second model is a semantic segmentation model, which is ultimately used to achieve pixel-level classification and recognition: identifying whether each pixel in the target skin area image corresponds to the pixel of the target object.

[0114] The following is a brief introduction to the training process of the second model.

[0115] First, a plurality of training sample images are obtained, wherein each training sample image is an image containing a target skin area. Taking the target skin area as the forehead skin area as an example, a plurality of face images containing the entire face area can be collected, and the forehead area image can be extracted from these face images as the training sample image.

[0116] Secondly, the labeling information corresponding to each of the plurality of training sample images is determined, and the labeling information includes the pixel position of the target object obtained by depicting the target object in the training sample image. Afterwards, the second model is trained until it converges according to the plurality of training sample images and the labeling information corresponding to each of the plurality of training sample images.

[0117] Whether in the process of training the second model or in the process of using the trained second model, the second model is used to identify the pixel position corresponding to the target object in the input image, wherein, in the training process, the input image is a training sample image, and in the process of use, the input image is a target skin area image extracted from the face image of the current user.

[0118] The following uses the second model as an example to illustrate how the second model identifies the pixel position of the target object in the target skin area image. Figure 5 shown.

[0119] Figure 5 A flowchart of a method for determining the pixel position of a target object in a target skin area image provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown, the method may include the following steps:

[0120] 501. Perform feature extraction on the target skin region image to obtain feature vectors of multiple different scales.

[0121] 502. Determine a plurality of prediction regions in the target skin region image, where the plurality of prediction regions include a plurality of selection boxes with different sizes set at different positions in the target skin region image.

[0122] 503. Predict multiple target prediction regions containing the target object among the multiple prediction regions according to the feature vectors of multiple scales.

[0123] 504. Extract local feature vectors corresponding to multiple target prediction regions from the feature vectors of multiple different scales.

[0124] 505. Generate second mask images corresponding to the multiple target prediction regions respectively according to the local feature vectors corresponding to the multiple target prediction regions; in the second mask image, pixels corresponding to the target object have a first gray value, and pixels corresponding to the non-target object have a second gray value.

[0125] 506. Determine pixel positions of the target object in the multiple target prediction regions according to the second mask images corresponding to the multiple target prediction regions respectively.

[0126] For ease of understanding, the following Figure 6 Some functions that the second model can provide are exemplified. In general, Figure 6 As shown in , the functions provided by the second model can be summarized as follows: feature extraction, position positioning, and pixel segmentation.

[0127] The purpose of position positioning is to locate the target skin area image containing the target object, usually a rectangular area. The purpose of pixel segmentation is to identify the pixels corresponding to the target object in the final located area. In simple terms, it is to classify and identify each pixel contained in the area to identify whether each pixel belongs to the target object or not.

[0128] The following specifically describes how to finally identify the pixel position of the target object in the target skin area image input into the second model through the above functions.

[0129] First, from the perspective of composition structure, the second model contains multiple network layers that provide feature extraction functions, and these multiple network layers constitute the backbone network of the second model. Through these multiple network layers, the target skin area image is subjected to feature extraction to obtain multiple feature vectors of different scales, such as Figure 6 The three scales of feature vectors from large to small are shown in Figure 1: P1, P2, and P3. The feature vector with a larger scale contains richer detail information because it corresponds to a smaller field of view, while the feature vector with a smaller scale contains more global information, i.e., high-level semantic information, because it corresponds to a larger field of view.

[0130] In practical applications, the above multiple network layers can form a convolutional network. Since the features extracted by the convolutional network are invariant to scaling, translation, rotation, etc., and have the advantages of strong feature expression ability and good generalization, they can be selected to extract image features. For example, multiple network layers can eventually form: ResNet network, MobileNet network, etc.

[0131] In addition, optionally, in order to obtain a more accurate target object pixel position positioning result, after obtaining the feature vectors of the above multiple scales, the feature vectors of the multiple scales may be fused. Figure 6 As shown in , for example, feature fusion processing is performed through feature pyramid networks (FPN) to obtain fused feature vectors of multiple different scales: P4, P5, and P6.

[0132] After performing the above feature extraction and fusion processing, the location area containing the target object can be predicted based on the obtained feature vectors of multiple scales.

[0133] In an optional embodiment, the position positioning function can be further divided into a rough position positioning function and a precise position positioning function. Based on this, optionally, the determination of the position area containing the target object can be implemented based on only the rough position positioning function. Optionally, the determination of the position area containing the target object can be implemented based on the rough position positioning function and the precise position positioning function.

[0134] Figure 6 The process shown in FIG. 1 is a process using only the coarse position positioning function.

[0135] The execution process of the coarse position positioning function is as follows:

[0136] First, multiple anchors with different sizes can be set at different positions in the target skin area image, and these anchors are used as multiple prediction areas that are initially set. The multiple anchors with different sizes set at different positions in the target skin area image can be: a plurality of pixel positions are determined in the target skin area image, and for each pixel position, a plurality of rectangular anchors with different sizes are set with the pixel position as the center, so that the multiple prediction areas are composed of the multiple anchors corresponding to the plurality of pixel positions. For ease of understanding, Figure 7 To illustrate this. Figure 7 In the example, take any two pixels in an image (the two black dots in the figure) as an example, and set three selection boxes of different sizes at the positions of these two pixels: K1, K2, and K3.

[0137] Afterwards, based on the obtained feature vectors of multiple scales (taking P4, P5, and P6 after feature fusion processing as an example), it is predicted whether the target object is contained in the above multiple prediction regions.

[0138] In summary, to achieve the rough location positioning function, the second model may include at least one network layer to predict whether the above-mentioned multiple prediction areas contain the target object through the at least one network layer. The at least one network layer may be implemented as a convolutional network.

[0139] like Figure 6 As shown in , the output result of the rough position positioning function may include two parameters: the position information (position) corresponding to each prediction area and its corresponding confidence (score). The position information is the position range corresponding to the corresponding prediction area in the target skin area image, and the confidence reflects the probability that the corresponding prediction area contains the target object. It can be considered that when the confidence corresponding to a prediction area is greater than the set threshold, it means that the prediction area contains the target object, otherwise, it does not contain it.

[0140] The following is a brief introduction on how to predict whether each prediction area contains the target object based on the rough location positioning function. This can be achieved through the following steps:

[0141] According to the sizes of the multiple prediction regions, target feature vectors corresponding to the multiple prediction regions are determined from feature vectors of multiple scales; and according to the target feature vectors corresponding to the multiple prediction regions, whether the corresponding prediction regions contain target objects is predicted.

[0142] The correspondence between the size of the prediction region and the scale of the feature vector is set in advance.

[0143] Assuming that the size of the prediction region includes the following three sizes from small to large: K1, K2, K3, assuming that the feature vectors of multiple scales that have been obtained are the following three feature vectors from large to small: P4, P5, P6, then the corresponding relationship between the size of the set prediction region and the scale of the characteristic vector can be: the prediction region of size K1 corresponds to the feature vector P4, the prediction region of size K2 corresponds to the feature vector P5, and the prediction region of size K3 corresponds to the feature vector P6. In other words, the prediction region of large size corresponds to the feature vector of small size, and the prediction region of small size corresponds to the feature vector of large size.

[0144] Based on the above assumptions, among all prediction areas, the target feature vectors corresponding to the prediction areas with a size of K1 are all feature vectors P4, the target feature vectors corresponding to the prediction areas with a size of K2 are all feature vectors P5, and the target feature vectors corresponding to the prediction areas with a size of K3 are all feature vectors P6.

[0145] Taking any prediction region i of size K1 as an example, whether the prediction region i contains the target object can be predicted based on the corresponding feature vector P4. Specifically, a local feature vector corresponding to the prediction region i (i.e., a part of the elements in the feature vector P4 corresponding to the prediction region i) can be determined from the feature vector P4, and whether the prediction region i contains the target object can be determined based on the local feature vector.

[0146] Through the above process, the prediction results corresponding to the multiple prediction regions can be obtained in the end, that is, whether the target object is included. Based on the prediction results, the prediction regions containing the target object can be determined to form multiple target prediction regions.

[0147] Afterwards, the local feature vectors corresponding to multiple target prediction regions are extracted from the feature vectors of multiple different scales through the following process:

[0148] Performing border regression processing on the multiple target prediction regions to obtain first position regions corresponding to the multiple target prediction regions;

[0149] Deduplication processing is performed on the first position regions corresponding to the plurality of target prediction regions to obtain a plurality of first position regions remaining after the deduplication processing;

[0150] A plurality of local feature vectors corresponding to each of the first position regions are extracted from the feature vectors of different scales.

[0151] Among them, the border regression processing can be implemented based on various border regression algorithms provided in the existing related technologies. In short, the result of the border regression is to adjust the position and size of the target prediction area so that the adjusted target prediction area can be closer to the location area where the target object exists. Here, the result of a target prediction area after border regression adjustment is called the first location area.

[0152] Since the first position regions of multiple target prediction regions after bounding box regression processing may overlap, after obtaining these first position regions, the obtained first position regions can be deduplicated, and the first position regions remaining after deduplication processing are called multiple first position regions. The deduplication processing can be implemented, for example, by using a non maximum suppression (NMS) algorithm, but of course, it is not limited thereto.

[0153] After obtaining the plurality of first position regions, local feature vectors corresponding to the plurality of first position regions are extracted from the feature vectors of the plurality of different scales obtained previously. Specifically, the target feature vectors corresponding to the plurality of first position regions can be determined from the feature vectors of the plurality of scales according to the respective sizes of the plurality of first position regions, so as to extract the local feature vectors corresponding to the plurality of first position regions from the target feature vectors corresponding to the plurality of first position regions. The execution of this process can participate in the acquisition process of the plurality of target prediction regions, which is similar and will not be described in detail.

[0154] Afterwards, since the sizes of the multiple first position regions are different, the scales of the local feature vectors corresponding to them are also different. After obtaining the local feature vectors corresponding to the multiple first position regions, as shown in FIG. Figure 6 As shown in , the local feature vectors corresponding to the multiple first position regions can be pooled (ROI Pooling) to normalize the local feature vectors corresponding to the multiple first position regions to the same scale. The scale after normalization can be customized.

[0155] In summary, the multiple first location areas are the determined location areas containing the target object.

[0156] Afterwards, based on the pixel segmentation function, determine whether each pixel in the multiple first position areas corresponds to the target object. Specifically, the second model may include at least one network layer capable of implementing the pixel segmentation function, and the at least one network layer may generate a second mask image corresponding to each of the multiple first position areas based on the normalized local feature vectors corresponding to each of the multiple first position areas. Finally, based on the second mask images corresponding to each of the multiple first position areas, determine whether each pixel in the multiple first position areas corresponds to the target object.

[0157] Taking any of the first position regions j as an example, the size of the second mask image generated corresponding thereto may not be equal to the size of the first position region j. Assuming that the size of the first position region j is 28*28, and the size of the second mask image is 14*14, then, after obtaining the second mask image, it is enlarged to the same size as the first position region j through the enlargement process. At this time, each pixel in the second mask image corresponds to each pixel in the first position region j one by one. Since the pixels corresponding to the target object in the second mask image are the first grayscale value (for example, the grayscale value 255 corresponding to white), and the pixels corresponding to the non-target object are the second grayscale value (for example, the grayscale value 0 corresponding to black), accordingly, each pixel corresponding to the first grayscale value is located in the first position region j, and these pixels are the pixels corresponding to the target object, and the other pixels in the first position region j are all pixels that do not correspond to the target object.

[0158] The above processing is performed on each first position area, and finally the pixel position corresponding to the target object in the target skin area image can be obtained.

[0159] The above describes the process of locating the pixel position corresponding to the target object in the target skin area image based on the feature extraction, rough position positioning and pixel segmentation functions provided by the second model.

[0160] As mentioned above, the positioning function provided by the second model can be divided into a coarse positioning function and a precise positioning function. Therefore, after obtaining multiple first position areas based on the above coarse positioning function, some position areas that are more likely to not correspond to the target object can be further filtered out based on the precise positioning function.

[0161] Combination Figure 8 To illustrate the process of locating the pixel position corresponding to the target object in the target skin area image based on the feature extraction, rough position positioning, precise position positioning and pixel segmentation functions provided by the second model.

[0162] like Figure 8 As shown in , after obtaining the normalized local feature vectors corresponding to the multiple first position areas through the process described above, the following steps can be performed based on the precise position positioning function:

[0163] Predicting each first position region containing the target object among the plurality of first position regions according to the normalized local feature vectors corresponding to each of the plurality of first position regions;

[0164] Performing frame regression processing on each first position region containing the target object to obtain a second position region corresponding to each first position region;

[0165] Performing deduplication processing on the second position areas corresponding to each first position area to obtain a plurality of second position areas remaining after the deduplication processing;

[0166] Determine target feature vectors corresponding to each of the plurality of second position areas from the feature vectors at the plurality of scales;

[0167] Extracting local feature vectors corresponding to each of the plurality of second position regions from the target feature vectors corresponding to each of the plurality of second position regions;

[0168] Second mask images corresponding to the plurality of second position regions are generated according to the local feature vectors corresponding to the plurality of second position regions.

[0169] Among them, Figure 8 As shown in , similar to the process of predicting multiple target prediction areas in the previous text, in the process of predicting each first position area containing the target object in the multiple first position areas, the position information (position) and the confidence (score) corresponding to each first position area are also output to determine each first position area containing the target object in the multiple first position areas based on the confidence. Based on this, the position area that is unlikely to contain the target object can be filtered out from the multiple first position areas.

[0170] After performing bounding box regression and deduplication processing on each of the determined first position regions containing the target object, the obtained position regions are referred to as a plurality of second position regions.

[0171] Among them, determining the target feature vectors corresponding to each of the multiple second position areas from the feature vectors of multiple scales can be specifically implemented as follows: determining the target feature vectors corresponding to each of the multiple second position areas from the feature vectors of multiple scales according to the respective sizes of the multiple second position areas.

[0172] In addition, if Figure 8 As shown in , in the process of generating the second mask images corresponding to the plurality of second position regions according to the local feature vectors corresponding to the plurality of second position regions, first, the local feature vectors corresponding to the plurality of second position regions are subjected to pooling processing (ROI Pooling) to normalize the local feature vectors corresponding to the plurality of second position regions to the same scale, and then, the second mask images corresponding to the plurality of second position regions are generated according to the normalized local feature vectors corresponding to the plurality of second position regions. So as to finally determine the pixel positions corresponding to the target object in the plurality of second position regions based on the second mask image.

[0173] In summary, based on the functions provided by the above-mentioned second model, the pixel position corresponding to the target object in the target skin area image can be located, so that based on the positioning result, interaction with the user can be achieved, such as highlighting the target object in the user's face image so that the user can intuitively see the positioning result.

[0174] In summary, the above solution can finally identify the characteristic information of the target object from the face image. In practical applications, based on the characteristic information of the target object, various interactions with the user can be achieved to improve the user experience.

[0175] For example, as mentioned above, the characteristic information of the target object may optionally include the pixel position of the target object. At this time, optionally, in order to enhance the interactivity with the user, the positioning effect of the target object may be displayed in the face image based on the pixel position of the identified target object. The positioning effect is, for example, to depict the target object with a line of a certain color, that is, to render the corresponding pixel into a certain specific color according to the pixel position of the target object. Based on this, the user can see the positioning result of the target object existing in the target skin area from the face image.

[0176] For another example, assuming that the characteristic information of the target object of the same user in different time periods has been obtained through the above scheme, when the characteristic information of the target object is detected again based on the face image of the user, the characteristic information of the target object obtained in history and the characteristic information of the target object currently obtained can be displayed in association to form a comparison. In addition, suitable products can be recommended to the user in combination with the changes in the characteristic information of the target object. For example, if the severity of a user's forehead wrinkles is found to be severe in February and moderate in March, then the wrinkle removal product recommended to the user in February is more effective and can continue to be recommended to the user. On the contrary, if it is found that the severity of the user's wrinkles has not been alleviated, other products can be recommended to the user.

[0177] In addition to the interactions exemplified above, targeted product recommendations can also be made to users based on the feature information of the target object.

[0178] Fig. 9 A flowchart of a commodity recommendation method provided by an embodiment of the present invention is shown in FIG. Fig. 9 As shown in , the product recommendation method may include the following steps:

[0179] 901. Collect a face image according to a user operation.

[0180] 902. Extract a target skin area image from the face image, and identify feature information of a target object in the target skin area image.

[0181] 903. Obtain recommended products based on the characteristic information of the target object, and display the recommended products on the terminal interface.

[0182] Among them, the target skin area image can be a forehead skin area image, the target object can be wrinkles, and the characteristic information of the target object can be the level information and / or pixel position of the wrinkles. At this time, the scheme can be used to detect relevant features of forehead wrinkles in face images.

[0183] The execution process of steps 901 and 902 may refer to the relevant descriptions in the aforementioned other embodiments and will not be repeated here.

[0184] Obtaining recommended products based on the characteristic information of the target object can be achieved as follows: pre-setting the correspondence between different characteristic information of the target object and the products, and based on this, filtering out recommended products corresponding to the characteristic information of the currently identified target object from the product database, and displaying these recommended products on the user's terminal interface for the user to select suitable products from them.

[0185] Specifically, as described above, the characteristic information of the target object may include level information of the target object, and the level information is one of a plurality of preset level information corresponding to the target object. Therefore, optionally, the recommended product may be obtained according to the level information of the target object.

[0186] For example, the target object is wrinkles, and the level information of wrinkles reflects the severity of wrinkles. Products corresponding to wrinkles of different severity levels can be pre-set, so as to obtain recommended products that match the wrinkle condition of the current user.

[0187] For another example, if the target object is acne, the level information of the acne can reflect the severity of the acne (for example, the severity is represented by the ratio of the acne area to the cheek area). Similarly, products corresponding to acne of different severities can be pre-set so as to obtain recommended products that match the current user's acne condition.

[0188] In addition, optionally, in the process of recommending matching products to users, in addition to combining the characteristic information of the target object, other factors identified from the face image may also be combined. For example, in the scenario of cosmetics recommendation, the user's skin brightness information may also be combined.

[0189] For ease of understanding, the execution process of the product recommendation method provided in this embodiment is exemplarily described below by taking the example of specifically recommending anti-wrinkle products to users.

[0190] In this exemplary application scenario, a target skin area of ​​a human face may be predefined, and the target skin area is a skin area prone to wrinkles, such as the forehead, the corners of the eyes, etc. Accordingly, the target object refers to wrinkles existing in the target skin area.

[0191] For ease of description, it is assumed that the target skin area is the forehead skin area and the target object is wrinkles. Fig.10 The anti-wrinkle product recommendation scenario shown illustrates the execution process of the above-mentioned product recommendation method from a macro perspective.

[0192] exist Fig.10 In the example, suppose a user wants to buy skin care products. The user can collect an image containing the entire face area of ​​the user, i.e., a face image, through a terminal device. The face image is uploaded to a server that maintains a large amount of product information, and the face image is used as an input image for a product recommendation method executed by the server. Specifically, the server can detect whether the user's forehead skin area contains wrinkles based on the product recommendation method, so as to recommend anti-wrinkle products to the user in a targeted manner based on the detection result.

[0193] After obtaining the face image, Fig.10As shown in , first, the server needs to extract the target skin area image, that is, the forehead skin area image, from the face image. In simple terms, the forehead skin area is identified from the face image, and the image corresponding to the skin area is cut out from the face image, which is called the forehead skin area image. Fig.10 In the example, it is assumed that the forehead skin area image includes Fig.10 Several wrinkles are indicated in the figure.

[0194] Afterwards, the server identifies characteristic information of wrinkles contained in the forehead skin area image, such as level information reflecting the severity of the wrinkles.

[0195] In actual applications, for the target object of wrinkles, multiple level information can be predefined, such as the following four level information: no wrinkles, mild wrinkles, moderate wrinkles, and severe wrinkles. Based on this, the level information of wrinkles identified by the server from the forehead skin area image can be one of the above multiple level information.

[0196] exist Fig.10 In the example, assuming that the level of wrinkles identified is severe wrinkles, anti-wrinkle products corresponding to the level of severe wrinkles can be determined based on the pre-set correspondence between different wrinkle levels and different anti-wrinkle products, and these anti-wrinkle products can be recommended to the user for selection.

[0197] Fig.11 A flowchart of a commodity recommendation method provided by an embodiment of the present invention is shown in FIG. Fig.11 As shown in , the product recommendation method may include the following steps:

[0198] 1101. Collect human body images according to user operations.

[0199] 1102. Extract a target part image from the human body image, and identify feature information of a target object in the target part image.

[0200] 1103. Obtain recommended products based on the characteristic information of the target object, and display the recommended products on the terminal interface.

[0201] Different from the above-mentioned other embodiments, in this embodiment, the product recommendation is not limited to the scenario of recommending products based on the characteristic information of the target object on the skin, but can also be based on the characteristic information of the target object on other parts of the human body. For example, the target object can be: teeth, hair, hands, etc.

[0202] For example, assume the following application scenario: recommend toothpaste and other tooth care products to users based on the color of their teeth. In this case, the collected human body image can be a face image, and in the face image, the user is in a state of opening his mouth and showing his teeth. The mouth image can be extracted from the face image, and then the color features of the teeth can be identified in the mouth image, so as to obtain the corresponding products based on the color features and recommend them to the user.

[0203] The process of extracting the mouth image can be simply described as first identifying a number of key points of the human face in the human image, including key points of eyes, eyebrows, nose, mouth, and other parts, and extracting the mouth image from the face image based on the identified mouth key points. The process of identifying tooth color in the mouth image can adopt the processing idea of ​​the second model introduced in the previous article, which will not be repeated here.

[0204] Assume the following application scenario: recommending hair care products to users based on their hair length and density. In this case, the captured human image may be an image including the head and upper body, and in the human image, the user may place his or her long hair in front of the body so that the length of his or her hair can be clearly reflected in the image. The image area corresponding to the hair can be extracted from the human image, and then the length and / or density features of the hair can be identified in the extracted image area, so as to obtain the corresponding product based on the identified features for recommendation to the user.

[0205] For the contents not introduced in detail in this embodiment, please refer to the relevant descriptions in other aforementioned embodiments. The implementation principles are similar and will not be repeated here.

[0206] The following will describe in detail the image detection device and the commodity recommendation device of one or more embodiments of the present invention. Those skilled in the art will appreciate that these image detection devices and commodity recommendation devices can be configured using commercially available hardware components through the steps taught in this solution.

[0207] Fig.12 A schematic diagram of the structure of an image detection device provided by an embodiment of the present invention is shown in FIG. Fig.12 As shown, the device includes: an acquisition module 11, an extraction module 12, and an identification module 13.

[0208] The acquisition module 11 is used to acquire a face image.

[0209] The extraction module 12 is used to extract the target skin area image from the face image.

[0210] The recognition module 13 is used to recognize feature information of a target object in the target skin area image.

[0211] Optionally, the extraction module 12 can be specifically used to: generate a first mask image corresponding to the facial image; in the first mask image, the pixels corresponding to the skin area are the first grayscale value, and the pixels corresponding to the non-skin area are the second grayscale value; determine a plurality of facial key points corresponding to the target skin area; based on the plurality of facial key points, determine in the first mask image a position range corresponding to the target skin area; and extract a target skin area image corresponding to the position range in the facial image.

[0212] Optionally, the feature information includes level information and / or pixel position; wherein the level information is one of a plurality of preset level information corresponding to the target object.

[0213] Optionally, the target object includes: wrinkles.

[0214] Optionally, the feature information includes the pixel position. In this case, the device further includes: an interaction module, configured to display a positioning effect of the target object in the face image according to the pixel position of the target object.

[0215] Optionally, the feature information includes the level information. In this case, the recognition module 13 can be specifically used to: input the target skin area image into the first model to output the level information of the target object in the target skin area image through the first model.

[0216] Optionally, the device also includes: a first training module, used to obtain multiple training sample images, wherein each training sample image is an image containing a target skin area; determine the annotation information corresponding to each of the multiple training sample images, the annotation information including the level information of the target object in the training sample image; and train the first model according to the multiple training sample images and the annotation information corresponding to each of the multiple training sample images.

[0217] Optionally, the feature information includes the pixel position. In this case, the recognition module 13 can be specifically used to: input the target skin area image into the second model to output the pixel position of the target object in the target skin area image through the second model.

[0218] Optionally, the device also includes: a second training module, used to obtain multiple training sample images, wherein each training sample image is an image containing a target skin area; determine the annotation information corresponding to each of the multiple training sample images, the annotation information including the pixel position of the target object obtained by depicting the target object in the training sample image; and train the second model based on the multiple training sample images and the annotation information corresponding to each of the multiple training sample images.

[0219] Optionally, in the process of determining the pixel position of the target object in the target skin area image, the recognition module 13 can be specifically used to: perform feature extraction on the target skin area image to obtain multiple feature vectors of different scales; determine multiple prediction areas in the target skin area image, the multiple prediction areas including multiple selection boxes of different sizes set at different positions in the target skin area image; predict multiple target prediction areas containing the target object in the multiple prediction areas based on the multiple feature vectors of different scales; extract local feature vectors corresponding to each of the multiple target prediction areas from the multiple feature vectors of different scales; generate second mask images corresponding to each of the multiple target prediction areas based on the local feature vectors corresponding to each of the multiple target prediction areas; in the second mask image, pixels corresponding to the target object are first grayscale values, and pixels corresponding to non-target objects are second grayscale values; determine the pixel position of the target object in the multiple target prediction areas based on the second mask images corresponding to each of the multiple target prediction areas.

[0220] Optionally, the device further includes: a feature fusion module, configured to perform feature fusion processing on the multiple feature vectors of different scales to obtain fused multiple feature vectors of different scales, wherein the fused multiple feature vectors of different scales are used to predict the target prediction area.

[0221] Optionally, in the process of predicting multiple target prediction areas that include the target object in the multiple prediction areas, the identification module 13 can be specifically used to: determine the target feature vectors corresponding to each of the multiple prediction areas from the feature vectors of the multiple scales according to the respective sizes of the multiple prediction areas; and predict whether the corresponding prediction area contains the target object according to the target feature vectors corresponding to each of the multiple prediction areas.

[0222] Optionally, in the process of extracting the local feature vectors corresponding to each of the multiple target prediction regions from the feature vectors of different scales, the identification module 13 can be specifically used to: perform border regression processing on the multiple target prediction regions to obtain the first position regions corresponding to each of the multiple target prediction regions; perform deduplication processing on the first position regions corresponding to each of the multiple target prediction regions to obtain the multiple first position regions remaining after deduplication processing; and extract the local feature vectors corresponding to each of the multiple first position regions from the feature vectors of different scales.

[0223] Optionally, in the process of extracting the local feature vectors corresponding to each of the multiple first position areas from the feature vectors of different scales, the identification module 13 can be specifically used to: determine the target feature vectors corresponding to each of the multiple first position areas from the feature vectors of the multiple scales according to the sizes of each of the multiple first position areas; and extract the local feature vectors corresponding to each of the multiple first position areas from the target feature vectors corresponding to each of the multiple first position areas.

[0224] Optionally, in the process of generating second mask images corresponding to each of the multiple target prediction areas according to the local feature vectors corresponding to each of the multiple target prediction areas, the recognition module 13 can be specifically used to include: performing pooling processing on the local feature vectors corresponding to each of the multiple first position areas to normalize the local feature vectors corresponding to each of the multiple first position areas to the same scale; generating second mask images corresponding to each of the multiple first position areas according to the normalized local feature vectors corresponding to each of the multiple first position areas.

[0225] Optionally, the recognition module 13 can also be used to: predict each first position area containing the target object among the multiple first position areas based on the normalized local feature vectors corresponding to each of the multiple first position areas; perform border regression processing on each first position area containing the target object to obtain a second position area corresponding to each first position area; perform deduplication processing on the second position areas corresponding to each first position area to obtain a plurality of second position areas remaining after deduplication processing; determine the target feature vector corresponding to each of the multiple second position areas from the feature vectors of the multiple scales; extract the local feature vector corresponding to each of the multiple second position areas from the target feature vectors corresponding to each of the multiple second position areas; and generate a second mask image corresponding to each of the multiple second position areas based on the local feature vectors corresponding to each of the multiple second position areas.

[0226] Among them, optionally, in the process of determining the target feature vectors corresponding to each of the multiple second position areas from the feature vectors of the multiple scales, the identification module 13 can be specifically used to: determine the target feature vectors corresponding to each of the multiple second position areas from the feature vectors of the multiple scales according to the respective sizes of the multiple second position areas.

[0227] Among them, optionally, in the process of generating second mask images corresponding to each of the multiple second position areas according to the local feature vectors corresponding to each of the multiple second position areas, the recognition module 13 can be specifically used to: perform pooling processing on the local feature vectors corresponding to each of the multiple second position areas to normalize the local feature vectors corresponding to each of the multiple second position areas to the same scale; generate second mask images corresponding to each of the multiple second position areas according to the normalized local feature vectors corresponding to each of the multiple second position areas.

[0228] Fig.12 The device shown can perform the aforementioned Figures 1 to 8 The image detection method provided in the illustrated embodiment, the detailed execution process and technical effects can be found in the description of the aforementioned embodiment, which will not be repeated here.

[0229] In one possible design, the above Fig.12 The structure of the image detection device shown can be implemented as an electronic device. Fig.13 As shown, the electronic device may include: a first processor 21, a first memory 22, and a first communication interface 23. The first memory 22 stores executable code, and when the executable code is executed by the first processor 21, the first processor 21 can at least implement the above-mentioned Figures 1 to 8 The image detection method provided in the illustrated embodiment.

[0230] In addition, an embodiment of the present invention provides a non-transitory machine-readable storage medium, wherein an executable code is stored on the non-transitory machine-readable storage medium. When the executable code is executed by a processor of an electronic device, the processor can at least implement the above-mentioned Figures 1 to 8 The image detection method provided in the illustrated embodiment.

[0231] Fig.14 A schematic diagram of a product recommendation device provided by an embodiment of the present invention is shown in FIG. Fig.14 As shown, the device includes: an image acquisition module 31, an image processing module 32, and a product recommendation module 33.

[0232] The image acquisition module 31 is used to acquire facial images according to user operations.

[0233] The image processing module 32 is used to extract a target skin area image from the face image and identify feature information of a target object in the target skin area image.

[0234] The product recommendation module 33 is used to obtain recommended products according to the characteristic information of the target object and display the recommended products on the terminal interface.

[0235] Optionally, the characteristic information includes level information, and the level information is one of a plurality of preset level information corresponding to the target object. Based on this, the product recommendation module 33 can be specifically used to obtain recommended products according to the level information of the target object.

[0236] Optionally, the feature information includes a pixel position. In this case, the device further includes: an interaction module, configured to display a positioning effect of the target object in the face image according to the pixel position of the target object.

[0237] Fig.14 The device shown can perform the aforementioned Figures 9 and 10 The detailed execution process and technical effects of the commodity recommendation method provided in the illustrated embodiment can be found in the description of the aforementioned embodiment and will not be repeated here.

[0238] In one possible design, the above Fig.14 The structure of the commodity recommendation device shown can be implemented as an electronic device. Fig.15 As shown, the electronic device may include: a second processor 41, a second memory 42, and a second communication interface 43. The second memory 42 stores executable code, and when the executable code is executed by the second processor 41, the second processor 41 can at least implement the above-mentioned Figures 9 and 10 The commodity recommendation method provided in the illustrated embodiment.

[0239] In addition, an embodiment of the present invention provides a non-transitory machine-readable storage medium, wherein an executable code is stored on the non-transitory machine-readable storage medium. When the executable code is executed by a processor of an electronic device, the processor can at least implement the above-mentioned Figures 9 and 10 The commodity recommendation method provided in the illustrated embodiment.

[0240] Fig.16 A schematic diagram of a product recommendation device provided by an embodiment of the present invention is shown in FIG. Fig.16 As shown, the device includes: an image acquisition module 51, an image processing module 52, and a product recommendation module 53.

[0241] The image acquisition module 51 is used to acquire human body images according to user operations.

[0242] The image processing module 52 is used to extract a target part image from the human body image and identify feature information of a target object in the target part image.

[0243] The product recommendation module 53 is used to obtain recommended products according to the characteristic information of the target object and display the recommended products on the terminal interface.

[0244] Fig.16The device shown can perform the aforementioned Fig.11 The detailed execution process and technical effects of the commodity recommendation method provided in the illustrated embodiment can be found in the description of the aforementioned embodiment and will not be repeated here.

[0245] In one possible design, the above Fig.16 The structure of the commodity recommendation device shown can be implemented as an electronic device. Fig.17 As shown, the electronic device may include: a third processor 61, a third memory 62, and a third communication interface 63. The third memory 62 stores executable code, and when the executable code is executed by the third processor 61, the third processor 61 can at least implement the above-mentioned Fig.11 The commodity recommendation method provided in the illustrated embodiment.

[0246] In addition, an embodiment of the present invention provides a non-transitory machine-readable storage medium, wherein an executable code is stored on the non-transitory machine-readable storage medium. When the executable code is executed by a processor of an electronic device, the processor can at least implement the above-mentioned Fig.11 The commodity recommendation method provided in the illustrated embodiment.

[0247] Fig.18 A flow chart of a beauty makeup method provided by an embodiment of the present invention is shown in FIG. Fig.18 As shown, the method may include the following steps:

[0248] 1801. Collect facial images.

[0249] 1802. Extract a target skin area image from the face image, and identify feature information of a target object in the target skin area image.

[0250] 1803. Display the beauty effect corresponding to the characteristic information of the target object.

[0251] Optionally, the target skin area image may be a forehead skin area image, the target object may be wrinkles, and its characteristic information may include level information and / or pixel position, wherein the level information is one of a plurality of preset level information. At this time, the characteristic information of forehead wrinkles in the face image may be detected through steps 1801-1802.

[0252] The detailed implementation process of steps 1801-1802 can be found in the relevant descriptions in other embodiments above and will not be repeated here.

[0253] The beauty makeup method provided in this embodiment can be executed by a terminal device on the user side, and the terminal device can be a virtual reality device (such as AR glasses). At this time, taking the detection of the characteristic information of forehead wrinkles as an example, the virtual reality device can display the beauty makeup effect corresponding to the characteristic information of forehead wrinkles. The beauty makeup effect can be a video animation.

[0254] Taking the detection of characteristic information of forehead wrinkles as an example, in summary, optionally, displaying a beauty effect corresponding to the characteristic information of the target object can be achieved by: determining a product with a concealer function corresponding to the characteristic information of the forehead wrinkles, and displaying the concealer effect and product information of the product.

[0255] Specifically, the corresponding product with concealing function can be determined according to the level information reflecting the severity of forehead wrinkles, and then, based on the pixel positions corresponding to the forehead wrinkles in the face image, a video animation and picture reflecting the concealing effect that may be obtained after using the product are generated and displayed, so that the user can intuitively understand the effect of using the product through the image.

[0256] Among them, any frame image constituting the video animation can be obtained in the following manner: copying a facial image, locating the pixel position corresponding to the forehead wrinkles in the copied facial image, and changing the grayscale value or transparency of the pixel corresponding to the pixel position to reflect the wrinkle concealing effect.

[0257] In addition, optionally, displaying the beauty effect corresponding to the characteristic information of the target object can also be realized by: determining a product with an anti-wrinkle function corresponding to the characteristic information of forehead wrinkles, and displaying the anti-wrinkle effect and product information of the product.

[0258] Similar to the concealing effect, the wrinkle-removing effect can also be reflected by video animation. In the video animation, the user can watch the wrinkle-removing effect that can be obtained after using the recommended wrinkle-removing product for a period of time.

[0259] The above-mentioned product information can be provided through text, voice, pictures, etc., and is used to introduce the brand, price, function, main ingredients and other information of the product.

[0260] Fig.19 A structural diagram of a beauty makeup device provided by an embodiment of the present invention is shown in FIG. Fig.19 As shown, the device includes: an image acquisition module 71, an image processing module 72, and a beauty display module 73.

[0261] The image acquisition module 71 is used to acquire facial images.

[0262] The image processing module 72 is used to extract a target skin area image from the face image and identify feature information of a target object in the target skin area image.

[0263] The beauty display module 73 is used to display the beauty effects corresponding to the characteristic information of the target object.

[0264] Optionally, the feature information includes level information and / or pixel position; wherein the level information is one of a plurality of preset level information corresponding to the target object.

[0265] Optionally, the target object includes: wrinkles.

[0266] Optionally, the beauty display module 73 can be specifically used to: determine a product with a blemish-covering function corresponding to the characteristic information of the target object; and display the blemish-covering effect of the product and product information.

[0267] Optionally, the beauty display module 73 can be specifically used to: determine a product with an anti-wrinkle function corresponding to the characteristic information of the target object; and display the anti-wrinkle effect of the product and product information.

[0268] Fig.19 The device shown can perform the aforementioned Fig.18 The detailed execution process and technical effects of the beauty makeup method provided in the illustrated embodiment can be found in the description of the aforementioned embodiment and will not be repeated here.

[0269] In one possible design, the above Fig.19 The structure of the beauty makeup device shown can be implemented as an electronic device. The electronic device can be a virtual reality device. Fig. 20 As shown, the electronic device may include: a fourth processor 81, a fourth memory 82, and a fourth communication interface 83. The fourth memory 82 stores executable code, and when the executable code is executed by the fourth processor 81, the fourth processor 81 can at least implement the above-mentioned Fig.18 The makeup method provided in the illustrated embodiment. Of course, the electronic device may also include a display screen.

[0270] In addition, an embodiment of the present invention provides a non-transitory machine-readable storage medium, wherein an executable code is stored on the non-transitory machine-readable storage medium. When the executable code is executed by a processor of an electronic device, the processor can at least implement the above-mentioned Fig.18 The beauty makeup method provided in the illustrated embodiment.

[0271] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.

[0272] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0273] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image detection method, comprising: Get face image; Extracting a target skin area image from the face image; Inputting the target skin area image into a second model, and outputting pixel positions including feature information of a target object in the target skin area image through the second model; The second model determines the pixel position of the target object in the target skin area image by the following steps: Performing feature extraction on the target skin area image to obtain feature vectors of multiple different scales; Determine a plurality of prediction areas in the target skin area image, wherein the plurality of prediction areas include a plurality of selection boxes with different sizes set at different positions in the target skin area image; Predicting, according to the multiple feature vectors of different scales, multiple target prediction regions including the target object in the multiple prediction regions; Extracting local feature vectors corresponding to each of the plurality of target prediction regions from the plurality of feature vectors of different scales; Generate second mask images corresponding to the plurality of target prediction regions respectively according to the local feature vectors corresponding to the plurality of target prediction regions; in the second mask image, pixels corresponding to the target object have a first grayscale value, and pixels corresponding to the non-target object have a second grayscale value; The pixel positions of the target object in the plurality of target prediction areas are determined according to the second mask images respectively corresponding to the plurality of target prediction areas.

2. The method according to claim 1, wherein extracting the target skin area image from the face image comprises: generating a first mask image corresponding to the face image; In the first mask image, pixels corresponding to the skin area have a first grayscale value, and pixels corresponding to the non-skin area have a second grayscale value; Determine a plurality of facial key points corresponding to a target skin region; Determining, in the first mask image, a position range corresponding to the target skin area according to the plurality of facial key points; A target skin area image corresponding to the position range is extracted from the face image.

3. The method according to claim 1, wherein the feature information further includes level information; in, The level information is one of a plurality of preset level information corresponding to the target object.

4. The method according to claim 1, wherein the target object comprises: wrinkle.

5. The method according to claim 1, further comprising: The positioning effect of the target object is displayed in the face image according to the pixel position of the target object.

6. The method according to claim 1, wherein the characteristic information further comprises level information; The method further comprises: The target skin area image is input into a first model, so as to output level information of a target object in the target skin area image through the first model.

7. The method according to claim 6, further comprising: Acquire multiple training sample images, wherein each training sample image is an image containing a target skin area; Determine the annotation information corresponding to each of the plurality of training sample images, wherein the annotation information includes level information of the target object in the training sample image; The first model is trained according to the multiple training sample images and the annotation information corresponding to each of the multiple training sample images.

8. The method according to claim 1, further comprising: Acquire multiple training sample images, wherein each training sample image is an image containing a target skin area; Determining the annotation information corresponding to each of the plurality of training sample images, the annotation information including a pixel position of a target object obtained by depicting the target object in the training sample image; The second model is trained according to the multiple training sample images and the annotation information corresponding to each of the multiple training sample images.

9. The method according to claim 1, further comprising: The feature vectors of the multiple scales are subjected to feature fusion processing to obtain fused feature vectors of the multiple scales. The fused feature vectors of the multiple scales are used to predict the target prediction area.

10. The method according to claim 1, wherein predicting a plurality of target prediction regions containing the target object among the plurality of prediction regions based on the plurality of feature vectors of different scales comprises: Determining, according to the sizes of the plurality of prediction regions, target feature vectors corresponding to the plurality of prediction regions from the feature vectors of the plurality of scales; According to the target feature vectors corresponding to the multiple prediction regions, it is predicted whether the corresponding prediction region contains the target object.

11. The method according to claim 10, wherein extracting local feature vectors corresponding to each of the plurality of target prediction regions from the plurality of feature vectors of different scales comprises: Performing frame regression processing on the multiple target prediction regions to obtain first position regions corresponding to the multiple target prediction regions respectively; Performing deduplication processing on the first position regions corresponding to the plurality of target prediction regions respectively, so as to obtain a plurality of first position regions remaining after the deduplication processing; The local feature vectors corresponding to each of the plurality of first position areas are extracted from the plurality of feature vectors of different scales.

12. The method according to claim 11, wherein extracting local feature vectors corresponding to each of the plurality of first position regions from the plurality of feature vectors of different scales comprises: Determining, according to the sizes of the first position regions, target feature vectors corresponding to the first position regions from the feature vectors of the multiple scales; From the target feature vectors corresponding to the plurality of first position areas, local feature vectors corresponding to the plurality of first position areas are extracted.

13. The method according to claim 12, wherein generating second mask images corresponding to each of the plurality of target prediction regions according to the local feature vectors corresponding to each of the plurality of target prediction regions comprises: Performing pooling processing on the local feature vectors corresponding to each of the multiple first position regions to normalize the local feature vectors corresponding to each of the multiple first position regions to the same scale; According to the normalized local feature vectors respectively corresponding to the plurality of first position regions, second mask images respectively corresponding to the plurality of first position regions are generated.

14. The method according to claim 13, wherein generating second mask images corresponding to each of the plurality of first position regions according to the normalized local feature vectors corresponding to each of the plurality of first position regions comprises: Predicting each first position region containing the target object among the multiple first position regions according to the normalized local feature vectors corresponding to each of the multiple first position regions; Performing frame regression processing on each first position area containing the target object to obtain a second position area corresponding to each first position area; Performing deduplication processing on the second position areas corresponding to the first position areas to obtain a plurality of second position areas remaining after the deduplication processing; Determine, from the feature vectors of the multiple scales, the target feature vectors corresponding to each of the multiple second position areas; Extracting local feature vectors corresponding to each of the plurality of second position regions from target feature vectors corresponding to each of the plurality of second position regions; According to the local feature vectors respectively corresponding to the plurality of second position regions, second mask images respectively corresponding to the plurality of second position regions are generated.

15. The method according to claim 14, wherein determining the target feature vectors corresponding to each of the plurality of second location regions from the feature vectors at the plurality of scales comprises: According to the sizes of the plurality of second position regions, target feature vectors corresponding to the plurality of second position regions are determined from the feature vectors of the plurality of scales.

16. The method according to claim 14, wherein generating second mask images corresponding to each of the plurality of second position regions according to the local feature vectors corresponding to each of the plurality of second position regions comprises: Performing pooling processing on the local feature vectors corresponding to each of the multiple second position regions to normalize the local feature vectors corresponding to each of the multiple second position regions to the same scale; According to the normalized local feature vectors respectively corresponding to the plurality of second position regions, second mask images respectively corresponding to the plurality of second position regions are generated.

17. An image detection device, comprising: An acquisition module, used for acquiring a face image; An extraction module, used to extract a target skin area image from the face image; A recognition module, used for inputting the target skin area image into a second model, and outputting pixel positions including characteristic information of a target object in the target skin area image through the second model; In the process of determining the pixel position of the target object in the target skin area image, the recognition module is specifically used to: perform feature extraction on the target skin area image to obtain multiple feature vectors of different scales; determine multiple prediction areas in the target skin area image, the multiple prediction areas include multiple selection boxes with different sizes set at different positions in the target skin area image; predict multiple target prediction areas containing the target object in the multiple prediction areas based on the multiple feature vectors of different scales; extract local feature vectors corresponding to each of the multiple target prediction areas from the multiple feature vectors of different scales; generate second mask images corresponding to each of the multiple target prediction areas based on the local feature vectors corresponding to each of the multiple target prediction areas; in the second mask image, pixels corresponding to the target object are first grayscale values, and pixels corresponding to non-target objects are second grayscale values; and determine the pixel positions of the target object in the multiple target prediction areas based on the second mask images corresponding to each of the multiple target prediction areas.

18. An electronic device, comprising: A memory, a processor and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the image detection method as described in any one of claims 1 to 16.

19. A non-transitory machine-readable storage medium having executable code stored thereon, and when the executable code is executed by a processor of an electronic device, the processor is caused to execute the image detection method as described in any one of claims 1 to 16.

20. A product recommendation method, comprising: Collecting facial images according to user operations; Extracting a target skin area image from the face image, and inputting the target skin area image into a second model, and outputting pixel positions including feature information of a target object in the target skin area image through the second model; Obtain recommended products based on the characteristic information of the target object, and display the recommended products on the terminal interface; The second model determines the pixel position of the target object in the target skin area image by the following steps: Performing feature extraction on the target skin area image to obtain feature vectors of multiple different scales; Determine a plurality of prediction areas in the target skin area image, wherein the plurality of prediction areas include a plurality of selection boxes with different sizes set at different positions in the target skin area image; Predicting, according to the multiple feature vectors of different scales, multiple target prediction regions including the target object in the multiple prediction regions; Extracting local feature vectors corresponding to each of the plurality of target prediction regions from the plurality of feature vectors of different scales; Generate second mask images corresponding to the plurality of target prediction regions respectively according to the local feature vectors corresponding to the plurality of target prediction regions; in the second mask image, pixels corresponding to the target object have a first grayscale value, and pixels corresponding to the non-target object have a second grayscale value; The pixel positions of the target object in the plurality of target prediction areas are determined according to the second mask images respectively corresponding to the plurality of target prediction areas.

21. The method according to claim 20, wherein the feature information further comprises level information, and the level information is one of a plurality of preset level information corresponding to the target object; The obtaining of recommended products according to the characteristic information of the target object includes: The recommended products are obtained according to the level information of the target object.

22. The method according to claim 20 or 21, further comprising: The positioning effect of the target object is displayed in the face image according to the pixel position of the target object.

23. A commodity recommendation device, comprising: An image acquisition module, used to acquire facial images according to user operations; An image processing module, used for extracting a target skin area image from the face image, inputting the target skin area image into a second model, and outputting pixel positions including feature information of a target object in the target skin area image through the second model; A product recommendation module, used to obtain recommended products based on the characteristic information of the target object and display the recommended products on the terminal interface; In the process of determining the pixel position of the target object in the target skin area image, the image processing module is specifically used to: perform feature extraction on the target skin area image to obtain multiple feature vectors of different scales; determine multiple prediction areas in the target skin area image, the multiple prediction areas include multiple selection boxes with different sizes set at different positions in the target skin area image; predict multiple target prediction areas containing the target object in the multiple prediction areas based on the multiple feature vectors of different scales; extract local feature vectors corresponding to each of the multiple target prediction areas from the multiple feature vectors of different scales; generate second mask images corresponding to each of the multiple target prediction areas based on the local feature vectors corresponding to each of the multiple target prediction areas; in the second mask image, pixels corresponding to the target object are first grayscale values, and pixels corresponding to non-target objects are second grayscale values; and determine the pixel positions of the target object in the multiple target prediction areas based on the second mask images corresponding to each of the multiple target prediction areas.

24. An electronic device comprising: A memory, a processor and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the product recommendation method as described in any one of claims 20 to 22.

25. A non-temporary machine-readable storage medium having executable code stored thereon, wherein when the executable code is executed by a processor of an electronic device, the processor executes the commodity recommendation method as described in any one of claims 20 to 22.

26. A product recommendation method, comprising: Collect human body images according to user operations; Extracting a target part image from the human body image, and inputting the target part image into a second model, and outputting pixel positions including feature information of a target object in the target part image through the second model; Obtain recommended products based on the characteristic information of the target object, and display the recommended products on the terminal interface; The second model determines the pixel position of the target object in the target part image by the following steps: Extracting features from the target part image to obtain feature vectors of multiple different scales; Determine a plurality of prediction regions in the target part image, wherein the plurality of prediction regions include a plurality of selection boxes with different sizes set at different positions in the target part image; Predicting, according to the multiple feature vectors of different scales, multiple target prediction regions including the target object in the multiple prediction regions; Extracting local feature vectors corresponding to each of the plurality of target prediction regions from the plurality of feature vectors of different scales; Generate second mask images corresponding to the plurality of target prediction regions respectively according to the local feature vectors corresponding to the plurality of target prediction regions; in the second mask image, pixels corresponding to the target object have a first grayscale value, and pixels corresponding to the non-target object have a second grayscale value; The pixel positions of the target object in the plurality of target prediction areas are determined according to the second mask images respectively corresponding to the plurality of target prediction areas.

27. A commodity recommendation device, comprising: An image acquisition module, used to acquire human body images according to user operations; An image processing module, used for extracting a target part image from the human body image, inputting the target part image into a second model, and outputting pixel positions including feature information of a target object in the target part image through the second model; A product recommendation module, used to obtain recommended products based on the characteristic information of the target object and display the recommended products on the terminal interface; In the process of determining the pixel position of the target object in the target part image, the image processing module is specifically used to: perform feature extraction on the target part image to obtain multiple feature vectors of different scales; determine multiple prediction areas in the target part image, the multiple prediction areas include multiple selection boxes with different sizes set at different positions in the target part image; predict multiple target prediction areas containing the target object in the multiple prediction areas according to the multiple feature vectors of different scales; extract local feature vectors corresponding to each of the multiple target prediction areas from the multiple feature vectors of different scales; generate second mask images corresponding to each of the multiple target prediction areas according to the local feature vectors corresponding to each of the multiple target prediction areas; in the second mask image, pixels corresponding to the target object are first grayscale values, and pixels corresponding to non-target objects are second grayscale values; and determine the pixel positions of the target object in the multiple target prediction areas according to the second mask images corresponding to each of the multiple target prediction areas.

28. An electronic device comprising: A memory, a processor and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the product recommendation method as described in claim 26.

29. A non-transitory machine-readable storage medium having executable code stored thereon, and when the executable code is executed by a processor of an electronic device, the processor is caused to execute the product recommendation method as described in claim 26.

30. A method of beauty care, comprising: Collecting facial images; Extracting a target skin area image from the face image, and inputting the target skin area image into a second model, and outputting pixel positions including feature information of a target object in the target skin area image through the second model; Displaying beauty effects corresponding to the characteristic information of the target object; The second model determines the pixel position of the target object in the target skin area image by the following steps: Performing feature extraction on the target skin area image to obtain feature vectors of multiple different scales; Determine a plurality of prediction areas in the target skin area image, wherein the plurality of prediction areas include a plurality of selection boxes with different sizes set at different positions in the target skin area image; Predicting, according to the multiple feature vectors of different scales, multiple target prediction regions including the target object in the multiple prediction regions; Extracting local feature vectors corresponding to each of the plurality of target prediction regions from the plurality of feature vectors of different scales; Generate second mask images corresponding to the plurality of target prediction regions respectively according to the local feature vectors corresponding to the plurality of target prediction regions; in the second mask image, pixels corresponding to the target object have a first grayscale value, and pixels corresponding to the non-target object have a second grayscale value; The pixel positions of the target object in the plurality of target prediction areas are determined according to the second mask images respectively corresponding to the plurality of target prediction areas.

31. The method according to claim 30, wherein the characteristic information further comprises level information; in, The level information is one of a plurality of preset level information corresponding to the target object.

32. The method according to claim 30, wherein the target object comprises: wrinkle.

33. According to the method of claim 32, the step of displaying the beauty effect corresponding to the characteristic information of the target object comprises: Determining a product with a blemish-covering function corresponding to the characteristic information of the target object; Display the concealer effect of the product and product information.

34. The method according to claim 32, wherein the step of displaying the beauty effect corresponding to the characteristic information of the target object comprises: Determining a product with a wrinkle-removing function corresponding to the characteristic information of the target object; The wrinkle-removing effect of the product and product information are displayed.

35. A beauty makeup device, comprising: An image acquisition module, used for acquiring facial images; An image processing module, used for extracting a target skin area image from the face image, inputting the target skin area image into a second model, and outputting pixel positions including feature information of a target object in the target skin area image through the second model; A beauty display module, used to display beauty effects corresponding to the characteristic information of the target object; In the process of determining the pixel position of the target object in the target skin area image, the image processing module is specifically used to: perform feature extraction on the target skin area image to obtain multiple feature vectors of different scales; determine multiple prediction areas in the target skin area image, the multiple prediction areas include multiple selection boxes with different sizes set at different positions in the target skin area image; predict multiple target prediction areas containing the target object in the multiple prediction areas based on the multiple feature vectors of different scales; extract local feature vectors corresponding to each of the multiple target prediction areas from the multiple feature vectors of different scales; generate second mask images corresponding to each of the multiple target prediction areas based on the local feature vectors corresponding to each of the multiple target prediction areas; in the second mask image, pixels corresponding to the target object are first grayscale values, and pixels corresponding to non-target objects are second grayscale values; and determine the pixel positions of the target object in the multiple target prediction areas based on the second mask images corresponding to each of the multiple target prediction areas.

36. An electronic device comprising: A memory, a processor and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the beauty method as described in any one of claims 30 to 34.

37. A non-transitory machine-readable storage medium having executable code stored thereon, and when the executable code is executed by a processor of an electronic device, the processor is caused to execute the beauty method as described in any one of claims 30 to 34.

Images