Image description information acquisition method, device, server and storage medium

Abstract

The present disclosure relates to a method, device, server, and storage medium for obtaining image description information. The method comprises: obtaining multiple images to be identified, and object description information for each image to be identified; determining a target object image in each image to be identified based on the object description information of each image to be identified; classifying the target object image in each image to be identified to obtain a target object classification set to which each image to be identified belongs; and determining image description information for each image to be identified based on the set description information of the target object classification set to which each image to be identified belongs. Compared to the related art method of manually defining a category system and then naming the classification categories, the present disclosure does not require a manually defined category system, and thus can improve the accuracy of the determined image names.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a method, device, server, and storage medium for acquiring image description information. Background Art

[0002] With the development of image processing technology, a technology for classifying objects captured in images has emerged. By manually pre-defining hierarchical categories of objects and identifying the objects in the image, the objects captured in the image can be classified according to the above hierarchical categories, thereby obtaining the classification information of the objects in the image as the image name used to describe the image.

[0003] In related technologies, classifying objects in an image requires a manually customized category system, followed by naming the classification categories to obtain the image name. However, the manually customized category system is relatively rough, and the accuracy of classifying objects in the image is low, so the accuracy of the determined image name is low. Summary of the Invention

[0004] The present disclosure provides a method, device, server, and storage medium for obtaining image description information to at least address the problem of low accuracy of image names determined in related technologies. The technical solutions of the present disclosure are as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a method for obtaining image description information is provided, comprising:

[0006] Acquire multiple images to be identified and object description information for each image to be identified;

[0007] Determining a target object image in each of the images to be identified based on the object description information of each of the images to be identified;

[0008] performing object classification on the target object images in the images to be identified to obtain a target object classification set to which the images to be identified belong;

[0009] The image description information of each image to be identified is determined according to the set description information of the target object classification set to which each image to be identified belongs.

[0010] In an exemplary embodiment, determining the target object image in each image to be identified based on the object description information of each image to be identified includes: determining a current image to be identified, obtaining the object image to be identified contained in the current image to be identified; and obtaining an object image to be identified that matches the object description information from the object image to be identified as the target object image.

[0011] In an exemplary embodiment, the object description information includes text information; obtaining the image of the object to be identified that matches the object description information from the image of the object to be identified as the target object image includes: inputting the image of the object to be identified and the object description information into a pre-trained image-text matching model, obtaining text features corresponding to the object description information and image features corresponding to the image of the object to be identified through the image-text matching model; obtaining a feature distance between the text features and the image features; and obtaining the target object image from the image of the object to be identified based on the feature distance.

[0012] In an exemplary embodiment, the object classification of the target object images in the images to be identified to obtain the target object classification set to which the images to be identified belong includes: inputting the target object images in the images to be identified into a pre-trained object classification model, and obtaining object classification features of the target object images through the object classification model; and performing image clustering processing on the target object images in the images to be identified according to the object classification features to obtain the target object classification set to which the images to be identified belong.

[0013] In an exemplary embodiment, the image clustering processing of the target object images in each image to be identified according to the fine-grained features includes: obtaining the feature similarity between the target object images in each image to be identified based on the object classification features; and performing image clustering processing on the target object images in each image to be identified according to the feature similarity.

[0014] In an exemplary embodiment, before determining the image description information of each image to be identified based on the set description information of the target object classification set to which each image to be identified belongs, it also includes: obtaining pre-set information attributes for generating set description information, and the combination relationship of each information attribute corresponding to the set description information; obtaining target description information matching the information attributes from the object description information of each image to be identified contained in the target object classification set; and combining the target description information according to the combination relationship to obtain the set description information.

[0015] In an exemplary embodiment, the object description information includes text information; the information attributes include: an entity object name attribute of the target object, an object brand attribute of the target object, and at least one of an object category attribute of the target object; when the information attributes include the entity object name attribute of the target object, the object brand attribute of the target object, and the object category attribute of the target object, obtaining target description information that matches the information attributes from the object description information of each image to be identified contained in the target object classification set includes: inputting the object description information of each image to be identified contained in the target object classification set into a pre-trained named entity recognition model, and obtaining first target description information that matches the entity object name attribute, second target description information that matches the object brand attribute, and third target description information that matches the object category attribute through the output of the named entity recognition model.

[0016] According to a second aspect of an embodiment of the present disclosure, there is provided an apparatus for acquiring image description information, comprising:

[0017] The recognition image acquisition unit is configured to acquire a plurality of images to be recognized and object description information for each image to be recognized;

[0018] a target object determining unit configured to determine a target object image in each of the images to be identified based on the object description information of each of the images to be identified;

[0019] a target object classification unit configured to perform object classification on the target object images in the images to be identified, and obtain a target object classification set to which the images to be identified belong;

[0020] The image description determining unit is configured to determine the image description information of each image to be identified based on the set description information of the target object classification set to which each image to be identified belongs.

[0021] In an exemplary embodiment, the target object determination unit is further configured to determine the current image to be identified, obtain the image of the object to be identified contained in the current image to be identified; and obtain the image of the object to be identified that matches the object description information from the image of the object to be identified as the target object image.

[0022] In an exemplary embodiment, the object description information includes text information; the target object determination unit is further configured to input the image of the object to be identified and the object description information into a pre-trained image-text matching model, obtain text features corresponding to the object description information and image features corresponding to the image of the object to be identified through the image-text matching model; obtain a feature distance between the text features and the image features; and obtain a target object image from the image of the object to be identified based on the feature distance.

[0023] In an exemplary embodiment, the target object classification unit is further configured to input the target object images in each image to be identified into a pre-trained object classification model, and obtain object classification features of the target object images through the object classification model; perform image clustering processing on the target object images in each image to be identified according to the object classification features, and obtain the target object classification set to which each image to be identified belongs.

[0024] In an exemplary embodiment, the target object classification unit is further configured to obtain feature similarities between the target object images in the images to be identified based on the object classification features; and perform image clustering processing on the target object images in the images to be identified according to the feature similarities.

[0025] In an exemplary embodiment, the image description determination unit is further configured to obtain pre-set information attributes for generating set description information, as well as the combination relationship of each information attribute corresponding to the set description information; obtain target description information matching the information attributes from the object description information of each image to be identified contained in the target object classification set; and combine the target description information according to the combination relationship to obtain the set description information.

[0026] In an exemplary embodiment, the object description information includes text information; the information attributes include: an entity object name attribute of the target object, an object brand attribute of the target object, and at least one of an object category attribute of the target object; when the information attributes include the entity object name attribute of the target object, the object brand attribute of the target object, and the object category attribute of the target object, the image description determination unit is further configured to input the object description information of each image to be identified contained in the target object classification set into a pre-trained named entity recognition model, and obtain first target description information matching the entity object name attribute, second target description information matching the object brand attribute, and third target description information matching the object category attribute through the output of the named entity recognition model.

[0027] According to a third aspect of an embodiment of the present disclosure, a server is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the method for obtaining image description information as described in any one of the embodiments of the first aspect.

[0028] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of a server, the server is enabled to execute the image description information acquisition method as described in any one of the embodiments of the first aspect.

[0029] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, wherein the computer program product includes instructions, and when the instructions are executed by a processor of a server, the server is able to execute the image description information acquisition method as described in any embodiment of the first aspect.

[0030] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0031] The method obtains multiple images to be identified and object description information for each image to be identified; determines the target object image in each image to be identified based on the object description information of each image to be identified; classifies the target object image in each image to be identified to obtain the target object classification set to which each image to be identified belongs; and determines the image description information of each image to be identified based on the set description information of the target object classification set to which each image to be identified belongs. The present invention determines the target object image from each image to be identified based on the object description information of the image to be identified, classifies the target object image to form a target object classification set, and uses the set description information of the target object classification set as the target object description information of the image to be identified. Compared with the method in the related art that requires an artificially customized category system and then names the classification categories, the present invention does not require an artificially customized category system, and thus can improve the accuracy of the determined image name.

[0032] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0034] Figure 1 The figure is a flowchart of a method for obtaining image description information according to an exemplary embodiment.

[0035] Figure 2 The flowchart of determining a target object image in each to-be-recognized image is shown according to an exemplary embodiment.

[0036] Figure 3 The flowchart of determining a target object image in each to-be-recognized image is shown according to another exemplary embodiment.

[0037] Figure 4 The flowchart of obtaining a classification set of target objects in each image to be identified is shown according to an exemplary embodiment.

[0038] Figure 5 The figure is a flowchart of obtaining set description information according to an exemplary embodiment.

[0039] Figure 6 It is a framework diagram of a commodity set construction algorithm according to an exemplary embodiment.

[0040] Figure 7 The figure is a block diagram of a device for acquiring image description information according to an exemplary embodiment.

[0041] Figure 8 The figure is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0042] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0043] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0044] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0045] Figure 1 is a flowchart of a method for obtaining image description information according to an exemplary embodiment. Figure 1As shown, the image recognition method is used in a server and includes the following steps.

[0046] In step S101 , a plurality of images to be identified and object description information for each image to be identified are obtained.

[0047] The server may be implemented as a standalone server or a server cluster consisting of multiple servers. The image to be identified refers to an image for which image description information recognition is required. The image may contain an object to be identified, i.e., a target object. The object description information refers to information used to describe the target object to be identified, such as text or voice information describing the target object. In this embodiment, the server may collect multiple images for which image description information recognition is required as images to be identified, and use information describing the target object in each image to be identified as the corresponding object description information.

[0048] For example, the image to be identified may refer to a product image of a product object that needs to be identified, and the object description information is the descriptive information used to describe the product object, which may be the product title corresponding to the product image. The server may pre-collect product images of a product object that needs to be identified, and the product titles corresponding to each product image, as multiple images to be identified, and object description information for each image to be identified.

[0049] In step S102 , a target object image in each image to be identified is determined based on the object description information of each image to be identified.

[0050] The target object image refers to the image corresponding to the target object area in the image to be identified. The image to be identified may contain more than one object, and the target object image refers to the image corresponding to the target object area in the image to be identified. For example, a product image to be identified may contain a hat, a piece of clothing, a skirt, etc., and if the product title corresponding to the product image is used to describe a hat, then the server can use the hat in the product image as the product to be identified, i.e., the target object, and the image corresponding to the hat area in the product image can be used as the target object image. In this embodiment, the server can determine the target object described by the object description information in each image to be identified based on the object description information of each image to be identified, and use the image area corresponding to the target object as the target object image.

[0051] In step S103, the target object images in each to-be-recognized image are classified to obtain a target object classification set to which each to-be-recognized image belongs.

[0052] The target object classification set refers to the classification set to which the target object image belongs after the target object is classified. That is, each target object classification set can contain target object images of the same classification. Specifically, after obtaining the target object image in each image to be identified in step S102, the target object contained in the target object image can be classified, for example, by implementing an object classification process using a classification model or classification algorithm, thereby obtaining multiple classification sets. Subsequently, the classification set to which each target object image belongs can be used as the target object classification set, so that the server can obtain the target object classification set corresponding to each image to be identified.

[0053] In step S104, image description information of each image to be identified is determined based on the set description information of the target object classification set to which each image to be identified belongs.

[0054] The set description information is information used to describe the target object classification set, for example, it can be the set name of the target object classification set, while the image description information refers to the description information used to describe each image to be identified, and can be the image name of each image to be identified. Specifically, in step S103, after the server determines the target object classification set corresponding to each image to be identified, it can also obtain the set name corresponding to each target object classification set as the set description information of the target object classification set. Thereafter, the above set description information can also be used as the image description information corresponding to the image to be identified contained in the target object classification set.

[0055] For example, the target object classification set can be a certain product set, which can contain multiple product images of the same type of products. The server can obtain the collection name corresponding to the product set and use the above collection name as the unified product name of the above same type of products, thereby obtaining the product image name corresponding to each product image, that is, the image description information corresponding to each image to be identified.

[0056] In the above-mentioned method for obtaining image description information, a plurality of images to be identified and object description information for each image to be identified are obtained; based on the object description information of each image to be identified, a target object image in each image to be identified is determined; the target object image in each image to be identified is subjected to object classification to obtain a target object classification set to which each image to be identified belongs; and based on the set description information of the target object classification set to which each image to be identified belongs, the image description information of each image to be identified is determined. The present disclosure determines the target object image from each image to be identified through the object description information of the image to be identified, classifies the target object image, forms a target object classification set, and uses the set description information of the target object classification set as the target object description information of the image to be identified. Compared with the method in the related art that requires an artificially customized category system and then names the classification categories, the present disclosure does not require an artificially customized category system, and thus can improve the accuracy of the determined image name.

[0057] In an exemplary embodiment, Figure 2 As shown, step S102 may further include:

[0058] In step S201, a current image to be recognized is determined, and an image of an object to be recognized contained in the current image to be recognized is obtained.

[0059] The current image to be identified refers to any one of the multiple images to be identified obtained by the server, and the object image to be identified refers to the area images corresponding to all the photographed objects contained in the current image to be identified. For example, a current image to be identified may contain a hat, clothes, and a skirt. The server can respectively obtain the hat area image, clothes area image, and skirt area image of the current image to be identified as the object image to be identified contained in the current image to be identified. In this embodiment, after the server obtains multiple images to be identified, the multiple images to be identified can be used as the current images to be identified, and the area images corresponding to all the photographed objects contained in each current image to be identified can be obtained as the object image to be identified contained in each current image to be identified.

[0060] For example, the server may input the collected multiple current images to be identified into a pre-trained object detection model, which may be a RetinaNet model for target detection. The model may use a pyramid structure to detect objects of different scales in the image, filter out different object areas contained in each current image to be identified, and then obtain each object image to be identified contained in the current image to be identified.

[0061] In step S202 , an image of an object to be identified that matches the object description information is obtained from the image of the object to be identified as a target object image.

[0062] After obtaining all the object images to be identified contained in the current image to be identified in step S201, the server can further filter out the object images to be identified that match the object description information from the above-mentioned object images to be identified based on the object description information corresponding to the current image to be identified, as the target object images corresponding to each current image to be identified.

[0063] In this embodiment, the image of the object to be identified contained in the image to be identified can be first detected, and then the image of the object to be identified that matches the object description information can be screened out as the final target object image. This can ensure the integrity of the image of the object to be identified and improve the accuracy of determining the target object image.

[0064] Furthermore, if Figure 3 As shown, the object description information includes text information; step S202 may further include:

[0065] In step S301, the image of the object to be identified and the object description information are input into a pre-trained image-text matching model, and the image-text matching model is used to obtain text features corresponding to the object description information and image features corresponding to the image of the object to be identified.

[0066] In this embodiment, the object description information may be information that describes the target object using text, for example, a paragraph of text information used to describe the target object. The image-text matching model refers to a pre-trained model for classifying and matching images according to text information, for example, a Chinese clipping, or CLIP, model. This model can extract text features of the text information and image features of the image to select images that match the text information. Specifically, in step S201, after the server obtains all the images of the objects to be identified contained in the current image to be identified and the object description information corresponding to the current image to be identified, the aforementioned images of the objects to be identified and the object description information may be input into the pre-trained image-text matching model, which then extracts the image features corresponding to the aforementioned images of the objects to be identified and the text features corresponding to the object description information.

[0067] In step S302, the feature distance between the text feature and the image feature is obtained;

[0068] In step S303, a target object image is acquired from the to-be-identified object image according to the feature distance.

[0069] Feature distance refers to the distance between text features and image features. In this embodiment, the trained image-text matching model can shorten the feature distance between the target object image and the object description information in the object image to be identified, and increase the feature distance between other object images other than the target object image and the object description information in the object image to be identified. Therefore, after obtaining the text features and image features through the image-text matching model in step S301, the image-text matching model can further obtain the text features corresponding to the object description information and the feature distance between the image features of each object image to be identified, and filter out the corresponding target object image from the above-mentioned object images to be identified based on the feature distance. For example, the object image to be identified with the smallest feature distance can be used as the target object image.

[0070] In this embodiment, after obtaining the image of the object to be identified, the trained image-text matching model can also be used to obtain the image features of each object image to be identified, as well as the text features corresponding to the object description information, so as to determine the target object image in the object image to be identified based on the distance between the features, thereby improving the accuracy of target object image recognition.

[0071] In an exemplary embodiment, Figure 4 As shown, step S103 may further include:

[0072] Step S401 : inputting the target object image in each to-be-recognized image into a pre-trained object classification model, and obtaining object classification features of the target object image through the object classification model.

[0073] Object classification features refer to the image feature expressions required for object classification and can be fine-grained features of an image. An object classification model refers to a pre-trained neural network model used to classify target objects corresponding to target object images. This model can classify different types of target objects. In this embodiment, the server can input the obtained target object images into a pre-trained object classification model, which extracts fine-grained features of each target object image as object classification features. The object classification model can be a fine-grained retrieval model, which can use constructed classification loss and ranking loss to train a convolutional neural network model. The classification loss can be used to perform separate fine-grained classification on each training sample image to achieve rapid model convergence, while the ranking loss enhances the fine-grained feature extraction capability of the fine-grained retrieval model by shortening the distance between sample images belonging to the same object and increasing the distance between sample images belonging to different objects. Specifically, after obtaining the target object image corresponding to each image to be identified, the server can input each target object image into the trained object classification model, which extracts object classification features for each target object image.

[0074] In step S402, image clustering processing is performed on the target object images in each image to be identified according to the object classification features to obtain the target object classification set to which each image to be identified belongs.

[0075] Afterwards, the server can implement image clustering processing of the target object images according to the fine-grained features, which can be clustering target object images with the same fine-grained features. For example, a density-based clustering algorithm, which can be a DBSCAN clustering algorithm, can be used to cluster target object images with the same object classification features, and generate multiple target object classification sets, as well as the target object classification sets to which the target object images contained in each image to be identified belong.

[0076] In this embodiment, after the server obtains the target object image in each image to be identified, the target object image can be input into a pre-trained object classification model, and the object classification features of each target object image can be mined through the model. The object classification features can also be used to achieve image clustering of the target object image, thereby improving the accuracy of image clustering and obtaining a more accurate target object classification set.

[0077] Furthermore, step S402 may further include: obtaining feature similarities between target object images in each image to be identified based on the object classification feature; and performing image clustering processing on the target object images in each image to be identified based on the feature similarities.

[0078] Feature similarity refers to the similarity between the object classification features corresponding to the target object images, for example, it can be represented by the feature distance between the object classification features. In this embodiment, after the server extracts the object classification features of each target object image through the object classification model, the feature similarity between the object classification features can be calculated respectively, and the target object images can be clustered by the feature similarity. For example, two target object images whose feature similarity is greater than a certain similarity threshold can be clustered to form a target object classification set.

[0079] In this embodiment, after the server determines the object classification features of each target object image, it can determine the feature similarity between each object classification feature respectively, and based on the above feature similarity, perform image clustering processing on the target object image, so that the clustered target object images can have similar object classification features, further improving the accuracy of target object image clustering.

[0080] In an exemplary embodiment, Figure 5 As shown, before step S204, the following steps may also be included:

[0081] Step S501: obtaining pre-set information attributes for generating set description information and a combination relationship between the information attributes corresponding to the set description information.

[0082] The set description information refers to information used to describe the target object classification set, and may refer to the set name of the target object classification set. The information attributes refer to the information attributes corresponding to the information required to generate the set description information, and the combination relationship of each information attribute refers to the combination relationship between the aforementioned information attributes. In this embodiment, the server may pre-set the information attributes of the information required to generate the set description information and the combination relationship between the aforementioned information attributes for the generated set description information.

[0083] Step S502 : acquiring target description information that matches the information attribute from the object description information of each image to be identified contained in the target object classification set.

[0084] The target description information refers to the description information that matches the above-mentioned information attributes in the object description information of each image to be identified contained in the target object classification set. In this embodiment, the generated set description information can be composed of the object description information corresponding to each image to be identified contained in the target object classification set. Since the object description information generally carries description information for describing different information attributes of the target object, the server can filter out the description information that matches the information attributes used to form the set description information from the above description information as the target description information.

[0085] For example, the target object classification set includes image A to be identified, image B to be identified, and image C to be identified, and the above-mentioned images to be identified correspond to object description information A, object description information B, and object description information C, respectively. Then the server can select from the above-mentioned object description information A, object description information B, and object description information C, where object description information A includes description information of information attribute A and information attribute B, object description information B includes description information of information attribute A and information attribute C, and object description information C includes description information of information attribute B and information attribute D. Then, if the information attributes determined for generating the set description information are information attribute A, information attribute C, and information attribute D, then the description information that matches the information attribute can be filtered out from the above-mentioned object description information as the target description information.

[0086] In step S503, the target description information is combined according to the combination relationship to obtain set description information.

[0087] Finally, after determining the target description information of different attributes, the server can combine the target description information obtained in step S502 according to a pre-set combination relationship, such as the order of multiple information attributes, to generate corresponding set description information.

[0088] For example, if the combination relationship is set to be a combination relationship of information attribute C-information attribute A-information attribute D, then after the server determines the target description information that matches the above information attributes, it can combine the target description information according to the order of each information attribute in the above combination relationship to obtain the corresponding set description information.

[0089] In this embodiment, it is also possible to filter out target description information that matches the information attributes from the object description information of each image to be identified contained in the target object classification set through pre-set information attributes and the combination relationship between the information attributes, and combine the target description information according to the above combination relationship, so that the obtained set description information has a unified expression form, thereby realizing standardized processing of the set description information.

[0090] Furthermore, the object description information includes text information; the information attributes include: the entity object name attribute of the target object, the object brand attribute of the target object, and at least one of the object category attributes of the target object; in the case where the information attributes include the entity object name attribute of the target object, the object brand attribute of the target object, and the object category attribute of the target object, step S502 may further include: inputting the object description information of each image to be identified contained in the target object classification set into a pre-trained named entity recognition model, and obtaining first target description information matching the entity object name attribute, second target description information matching the object brand attribute, and third target description information matching the object category attribute through the output of the named entity recognition model.

[0091] In this embodiment, the object description information can be text information, and the target object classification set is a set of goods composed of goods of the same type. The set description information generated for the set of goods, that is, the set name of the formed set of goods, can be composed of the entity object name of the goods, the brand name of the goods, and the category attributes of the goods. Therefore, the entity object name attribute of the goods, the object brand attribute of the goods, and the object category attribute of the goods can be used as information attributes for generating the set description information.

[0092] At the same time, the named entity recognition model is used to identify the information attributes corresponding to the description information and the description information corresponding to the information attributes. The named entity recognition model can be implemented by a named entity recognition model based on language representation, that is, a NER model based on Bert. Specifically, if the information attributes include the entity object name attribute of the target object, the object brand attribute of the target object, and the object category attribute of the target object, the server can input the object description information of each image to be identified contained in the target object classification set into the above-mentioned named entity recognition model, and then the named entity recognition model can output the description information matching the entity object name attribute for the target object classification set, that is, the first target description information, the description information matching the object brand attribute, that is, the second target description information, and the description information matching the object category attribute, that is, the third target description information.

[0093] In this embodiment, a pre-trained named entity recognition model can be used to identify first target description information that matches the entity object name attribute, second target description information that matches the object brand attribute, and third target description information that matches the object category attribute. Therefore, set description information can be generated based on the above-mentioned first target description information, second target description information, and third target description information, so that standardized set description information for commodity sets can be obtained, further improving the standardization of the commodity set name.

[0094] In an exemplary embodiment, a method for constructing a product set based on multimodal understanding is also provided, which mainly includes three parts: (1) detection and identification of intended products; 2) aggregation of products of the same type into a product set; and 3) automatic naming of the product set, including brand identification, key attribute identification, and entity name identification. In the detection and identification of intended products, it is necessary to detect all products in the main image, perform cross-modal matching between the detection frame and the product title to obtain the intended products and put them into the warehouse, while non-intended products are removed as noise. Since different merchants may sell the same product, it is necessary to perform fine-grained same-type retrieval on the intended products and aggregate all the products of the same type into a product set as a unified information expression of the products. At the same time, different merchants have different styles of naming when listing products, and in order to increase the probability of the products being hit by the search system, merchants will consciously extend the length of the title, and even add some irrelevant adjectives or nouns. Therefore, it is also necessary to automatically and accurately name the product set, and obtain the real and standard naming of the product set by mining its brand name, key attributes, and entity name through graphics and text. The above-mentioned product collection construction method mainly includes three core technical modules, which involve object detection algorithm, cross-modal matching algorithm, fine-grained same-item retrieval algorithm, entity naming recognition algorithm, etc.

[0095] like Figure 6 As shown in the figure, this method mainly includes three core modules:

[0096] Intended product detection: The RetinaNet detection model is used to detect all products in the main product image. The CLIP model is then trained to extract the spatial feature representations of the product title and the product detection box. The intended product is determined based on the degree of match between the title and product features.

[0097] Aggregation of similar products. A fine-grained similar product retrieval CNN model is trained to extract more refined feature representations of the intended products. Similar products are then aggregated into a product set through feature clustering.

[0098] Product collections are named and stored. We perform NER entity parsing on all titles in the product collection to obtain the brand name + key attribute name + entity name, and combine them as the standardized name of the products in the product collection.

[0099] (1) Intended product detection:

[0100] The first step in detecting intended products is to obtain detection frames for all products. This embodiment uses the RetinaNet model for product detection. RetinaNet uses a pyramid structure to better detect objects of different scales in the main image and proposes Focal Loss to balance the imbalance between positive and negative samples and the imbalance between easy and difficult samples in the detection problem. The RetinaNet model can effectively detect all products in the main image, such as shoes, bags, cosmetics, clothing, home appliances, toys, books, food, jewelry, furniture, etc. However, a main product image only sells one product, so all products detected in the main image need to be denoised to retain only the intended product.

[0101] The second step in detecting intended products is cross-modal matching of images and text to obtain the intended product frames. The Chinese CLIP model is trained to obtain the same spatial feature representations of the product title and the main image frame. For example, a title might read "Pleated Skirt Summer High-Waisted Miniskirt XX Preppy Style Black Plus Size Skirt Gray," and the main image might contain frames for shoes, skirts, and tops. The core idea of ​​the CLIP model is to narrow the feature distance between the title and the skirt frame, while widening the feature distance between the title and other product frames. This is achieved through contrastive learning.

[0102] (2) Aggregation of similar products:

[0103] The goal of product aggregation is to aggregate the same product sold by different merchants into a single product collection. This requires fine-grained differentiation of products, such as different styles of clothing. A fine-grained retrieval model is proposed to exploit the fine-grained feature expression capabilities of product bounding boxes. Classification and ranking losses are used to comprehensively supervise the training of the CNN network. The classification loss primarily performs fine-grained classification on each product, facilitating rapid model convergence. The ranking loss uses a triplet loss function to narrow the distance between products of the same style and distance between different styles, further enhancing the CNN network's mining capabilities.

[0104] After optimizing the same-item retrieval model, we can obtain the feature expression of the product intent product box. Based on this feature expression, we use the density-based clustering algorithm, DBSCAN, to cluster the same items. This clustering algorithm can easily aggregate the same items into a cluster. Compared with the commonly used Kmeans clustering algorithm, DBSCAN does not require manual specification of the number of clusters and can tolerate noisy samples well.

[0105] In order to further optimize the aggregation effect, the above two steps can be performed iteratively. After clustering, the labels of the product set are obtained, and the product set labels are used to optimize the retrieval model, thereby optimizing the new clustering effect and guiding the basic convergence of the clustering clusters, that is, obtaining the product information at the final product set granularity.

[0106] (3) NER parsing of brand attributes and entity names

[0107] After obtaining product information at the product set granularity, named entity recognition (NER) parsing is needed to generate standardized naming for the product set, using a combination of brand name, key attribute name, and entity name. For example, for a "Pleated Skirt Summer High-Waisted Short Skirt XX Style Black Plus-Size Skirt Gray Brand A," the product set to be stored is named: Brand A [Skirt] Pleated Skirt, where "Brand A" is the brand, "Skirt" is the key attribute (optional), and "Pleated Skirt" is the entity name. To achieve this, a Bert-based NER model is trained to simultaneously identify the brand name, key attribute name, and entity name.

[0108] Through the above embodiments, the manual definition of the construction of the commodity collection category system can be avoided, thereby avoiding the coarseness of the category system granularity, and the collection naming does not rely on manual definition. At the same time, the generated commodity collection can describe more accurate commodity information than the manually defined three-level category system.

[0109] It should be understood that although Figure 1-Figure 5 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1-Figure 5 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The order of execution of these steps or stages is not necessarily one by one, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0110] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can be referred to each other, and each embodiment focuses on the differences from other embodiments. For related parts, please refer to the description of other method embodiments.

[0111] Figure 7 FIG. 1 is a block diagram of an image recognition device according to an exemplary embodiment. Figure 7 The device includes a recognition image acquisition unit 701, a target object determination unit 702, a target object classification unit 703 and an image description determination unit 704.

[0112] The recognition image acquisition unit 701 is configured to acquire a plurality of images to be recognized and object description information for each image to be recognized;

[0113] The target object determining unit 702 is configured to determine the target object image in each image to be identified based on the object description information of each image to be identified;

[0114] The target object classification unit 703 is configured to perform object classification on the target object images in each to-be-identified image to obtain a target object classification set to which each to-be-identified image belongs;

[0115] The image description determining unit 704 is configured to determine the image description information of each image to be identified based on the set description information of the target object classification set to which each image to be identified belongs.

[0116] In an exemplary embodiment, the target object determination unit 702 is further configured to determine the current image to be identified, obtain the image of the object to be identified contained in the current image to be identified, and obtain the image of the object to be identified that matches the object description information from the image of the object to be identified as the target object image.

[0117] In an exemplary embodiment, the object description information includes text information; the target object determination unit 702 is further configured to input the image of the object to be identified and the object description information into a pre-trained image-text matching model, obtain text features corresponding to the object description information and image features corresponding to the image of the object to be identified through the image-text matching model; obtain the feature distance between the text features and the image features; and obtain the target object image from the image of the object to be identified based on the feature distance.

[0118] In an exemplary embodiment, the target object classification unit 703 is further configured to input the target object image in each image to be identified into a pre-trained object classification model, and obtain the object classification features of the target object image through the object classification model; perform image clustering processing on the target object image in each image to be identified according to the object classification features, and obtain the target object classification set to which each image to be identified belongs.

[0119] In an exemplary embodiment, the target object classification unit 703 is further configured to obtain feature similarities between target object images in each image to be identified based on object classification features; and perform image clustering processing on the target object images in each image to be identified according to the feature similarities.

[0120] In an exemplary embodiment, the image description determination unit 704 is further configured to execute the acquisition of pre-set information attributes for generating set description information, and the combination relationship of each information attribute corresponding to the set description information; obtain target description information matching the information attributes from the object description information of each image to be identified contained in the target object classification set; and combine the target description information according to the combination relationship to obtain the set description information.

[0121] In an exemplary embodiment, the object description information includes text information; the information attributes include: the entity object name attribute of the target object, the object brand attribute of the target object, and at least one of the object category attributes of the target object; when the information attributes include the entity object name attribute of the target object, the object brand attribute of the target object, and the object category attribute of the target object, the image description determination unit 704 is further configured to input the object description information of each image to be identified contained in the target object classification set into a pre-trained named entity recognition model, and obtain first target description information matching the entity object name attribute, second target description information matching the object brand attribute, and third target description information matching the object category attribute through the output of the named entity recognition model.

[0122] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0123] Figure 8 FIG. 8 is a block diagram of an electronic device 800 for image recognition according to an exemplary embodiment. For example, the electronic device 800 may be a server. Figure 8 The electronic device 800 includes a processing component 820, which further includes one or more processors, and a memory resource represented by a memory 822 for storing instructions executable by the processing component 820, such as an application. The application stored in the memory 822 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 820 is configured to execute the instructions to perform the above method.

[0124] The electronic device 800 may further include a power supply component 824 configured to perform power management of the electronic device 800, a wired or wireless network interface 826 configured to connect the electronic device 800 to a network, and an input / output (I / O) interface 828. The electronic device 800 may operate based on an operating system stored in the memory 822, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or the like.

[0125] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 822 including instructions, and the instructions can be executed by a processor of the electronic device 800 to perform the above method. The storage medium can be a computer-readable storage medium, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0126] In an exemplary embodiment, a computer program product is further provided. The computer program product includes instructions, and the instructions can be executed by a processor of the electronic device 800 to implement the above method.

[0127] It should be noted that the above-mentioned devices, electronic devices, computer-readable storage media, computer program products, etc. can also include other implementation methods according to the description of the method embodiments. The specific implementation methods can refer to the description of the relevant method embodiments and will not be described one by one here.

[0128] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

[0129] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for obtaining image description information, characterized in that: include: Acquire multiple images to be identified and object description information for each image to be identified; Determining a target object image in each of the images to be identified based on the object description information of each of the images to be identified; performing object classification on the target object images in the images to be identified to obtain a target object classification set to which the images to be identified belong; Determining image description information of each image to be identified based on set description information of the target object classification set to which each image to be identified belongs; Before determining the image description information of each image to be identified based on the set description information of the target object classification set to which each image to be identified belongs, the method further includes: Obtaining pre-set information attributes for generating set description information, and a combination relationship between the information attributes corresponding to the set description information; Obtaining target description information that matches the information attribute from the object description information of each image to be identified contained in the target object classification set; The target description information is combined according to the combination relationship to obtain the set description information.

2. The method according to claim 1, characterized in that The step of determining the target object image in each of the images to be identified based on the object description information of each of the images to be identified includes: Determine a current image to be recognized, and obtain an image of an object to be recognized contained in the current image to be recognized; An image of the object to be identified that matches the object description information is obtained from the image of the object to be identified as the target object image.

3. The method according to claim 2, characterized in that The object description information includes text information; The step of acquiring, from the image of the object to be identified, an image of the object to be identified that matches the object description information as the target object image includes: Inputting the image of the object to be identified and the object description information into a pre-trained image-text matching model, and obtaining text features corresponding to the object description information and image features corresponding to the image of the object to be identified through the image-text matching model; Obtaining a feature distance between the text feature and the image feature; A target object image is acquired from the to-be-identified object image according to the feature distance.

4. The method according to claim 1, wherein The performing object classification on the target object images in the images to be identified to obtain a target object classification set to which the images to be identified belong includes: Inputting the target object image in each of the images to be identified into a pre-trained object classification model, and obtaining object classification features of the target object image through the object classification model; Image clustering processing is performed on the target object images in the images to be identified according to the object classification features to obtain a target object classification set to which the images to be identified belong.

5. The method according to claim 4, characterized in that The performing image clustering processing on the target object images in the images to be identified according to the object classification features includes: Obtaining feature similarities between target object images in each of the images to be identified based on the object classification features; Image clustering processing is performed on the target object images in the images to be identified according to the feature similarity.

6. The method according to claim 1, wherein The object description information includes text information; the information attributes include: at least one of the entity object name attribute of the target object, the object brand attribute of the target object, and the object category attribute of the target object; In a case where the information attributes include an entity object name attribute of the target object, an object brand attribute of the target object, and an object category attribute of the target object, obtaining target description information matching the information attributes from the object description information of each to-be-recognized image included in the target object classification set includes: The object description information of each image to be identified contained in the target object classification set is input into a pre-trained named entity recognition model, and the named entity recognition model outputs first target description information that matches the entity object name attribute, second target description information that matches the object brand attribute, and third target description information that matches the object category attribute.

7. An image description information acquisition device, characterized in that: include: The recognition image acquisition unit is configured to acquire a plurality of images to be recognized and object description information for each image to be recognized; a target object determining unit configured to determine a target object image in each of the images to be identified based on the object description information of each of the images to be identified; a target object classification unit configured to perform object classification on the target object images in the images to be identified, and obtain a target object classification set to which the images to be identified belong; an image description determining unit configured to determine image description information of each image to be identified based on set description information of a target object classification set to which each image to be identified belongs; The image description determination unit is further configured to execute acquisition of pre-set information attributes for generating set description information, and the combination relationship of each information attribute corresponding to the set description information; obtain target description information matching the information attributes from the object description information of each image to be identified contained in the target object classification set; and combine the target description information according to the combination relationship to obtain the set description information.

8. The device according to claim 7, characterized in that The target object determination unit is further configured to determine a current image to be identified, obtain an image of an object to be identified contained in the current image to be identified, and obtain an image of an object to be identified that matches the object description information from the image of the object to be identified as the target object image.

9. The device according to claim 8, characterized in that The object description information includes text information; the target object determination unit is further configured to input the image of the object to be identified and the object description information into a pre-trained image-text matching model, obtain text features corresponding to the object description information and image features corresponding to the image of the object to be identified through the image-text matching model; obtain a feature distance between the text features and the image features; and obtain a target object image from the image of the object to be identified based on the feature distance.

10. The device according to claim 7, characterized in that The target object classification unit is further configured to input the target object image in each of the images to be identified into a pre-trained object classification model, and obtain object classification features of the target object image through the object classification model; Image clustering processing is performed on the target object images in the images to be identified according to the object classification features to obtain a target object classification set to which the images to be identified belong.

11. The device according to claim 10, characterized in that The target object classification unit is further configured to obtain feature similarities between the target object images in the images to be identified based on the object classification features; and perform image clustering processing on the target object images in the images to be identified according to the feature similarities.

12. The device according to claim 7, characterized in that The object description information includes text information; the information attributes include: the entity object name attribute of the target object, the object brand attribute of the target object, and at least one of the object category attributes of the target object; when the information attributes include the entity object name attribute of the target object, the object brand attribute of the target object, and the object category attribute of the target object, the image description determination unit is further configured to input the object description information of each image to be identified contained in the target object classification set into a pre-trained named entity recognition model, and obtain first target description information matching the entity object name attribute, second target description information matching the object brand attribute, and third target description information matching the object category attribute through the output of the named entity recognition model.

13. A server, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method for obtaining image description information according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of a server, the server is enabled to execute the method for obtaining image description information according to any one of claims 1 to 6.

15. A computer program product comprising instructions, characterized in that: When the instruction is executed by a processor of the server, the server is enabled to execute the method for obtaining image description information according to any one of claims 1 to 6.

Images