Article aggregation method and device, electronic equipment and readable storage medium
By extracting and fusing the image and text features of the product, using density clustering algorithms, the problems of low retrieval efficiency and poor scalability of similar items in the prior art are solved, and efficient and automatic determination of similar items are achieved.
Patent Information
- Application Number
- CN202311369638.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-20
- Publication Date
- 2025-04-22
AI Technical Summary
In the prior art, the search system based on product pictures is inefficient, and the matching scheme based on product attributes is poorly scalable, which requires high labor costs.
By obtaining the image data and text data of the target item, the image encoder model and the text encoder model are used to extract the image representation vector and the text representation vector respectively, and fuse these vectors to obtain the fusion representation vector, and similar items are determined based on the density clustering algorithm.
It realizes efficiently determining similar items of target items, without manual annotation, and has strong scalability.
Smart Images
Figure CN119862425A_ABST
Abstract
Description
Background Art
[0002] Identifying similar items from a massive pool of items is a very important basic technical capability, which can be used for competitive product detection, product recommendations, search result display, etc.; in related technologies, the retrieval system based on product images only considers the similarity of product images. Since the product images uploaded by different merchants may be quite different, the number of similar products recalled based on image features is small; the matching scheme based on product attributes needs to determine the key attributes for different categories of products and perform attribute normalization processing. The scheme has poor scalability and requires high labor costs.
[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0004] The present disclosure provides an item aggregation method, device, electronic device and computer-readable storage medium, which at least to some extent overcome the problem of low efficiency in determining similar items in the related art.
[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.
[0006] According to one aspect of the present disclosure, there is provided an object aggregation method, comprising: acquiring target image data and target text data corresponding to a target object; inputting the target image data into an image encoder model to obtain a target image representation vector; inputting the target text data into a text encoder model to obtain a target text representation vector; obtaining a target fusion representation vector based on the target image representation vector and the target text representation vector; and determining similar objects corresponding to the target object based on the target fusion representation vector based on a density clustering algorithm.
[0007] In one embodiment of the present disclosure, the method of determining similar items corresponding to the target item based on the target fusion representation vector based on the density clustering algorithm includes: determining the cluster cluster corresponding to the target fusion representation vector based on the target fusion representation vector and feature pool data based on the density clustering algorithm; and determining the similar items corresponding to the target item based on the cluster cluster.
[0008] In one embodiment of the present disclosure, the density clustering algorithm includes: a density-based clustering method with noise or an algorithm for identifying clustering structure by sorting sample points.
[0009] In one embodiment of the present disclosure, it also includes: determining one or more negative sample items corresponding to the positive sample item based on the positive sample item; determining a text loss function based on the text data of the positive sample item and the negative sample item; and training the text encoder model based on the text loss function.
[0010] In one embodiment of the present disclosure, determining the text loss function based on the positive sample items and the negative sample items includes: inputting the positive text data of the positive sample items into the text encoder model to obtain a plurality of positive text representation vectors; inputting the negative text data of the negative sample items into the text encoder model to obtain a plurality of negative text representation vectors; and determining the text loss function based on the positive text representation vectors and the negative text representation vectors.
[0011] In one embodiment of the present disclosure, it also includes: determining one or more negative sample items corresponding to the positive sample item based on the positive sample item; determining an image loss function based on the image data of the positive sample item and the negative sample item; and training the image encoder model based on the image loss function.
[0012] In one embodiment of the present disclosure, determining the image loss function based on the positive sample object and the negative sample object includes: inputting a plurality of positive image data of the positive sample object into the image encoder model to obtain a plurality of positive image representation vectors; inputting a plurality of negative image data of the negative sample object into the image encoder model to obtain a plurality of negative image representation vectors; and determining the image loss function based on the positive image representation vectors and the negative image representation vectors.
[0013] In one embodiment of the present disclosure, it also includes: obtaining the multiple positive image data based on the positive sample objects based on multiple data enhancement methods; and obtaining the multiple negative image data based on the negative sample objects based on multiple data enhancement methods.
[0014] In one embodiment of the present disclosure, the data enhancement method includes at least one of the following: random cropping, horizontal or vertical flipping, random rotation, Gaussian blur, grayscale transformation, brightness transformation, contrast adjustment, and saturation adjustment.
[0015] In one embodiment of the present disclosure, the image encoder model is a picture encoder model or a video encoder model.
[0016] In one embodiment of the present disclosure, obtaining a target fusion representation vector based on the target image representation vector and the target text representation vector includes: obtaining a fusion weight value; and obtaining the target fusion representation vector based on the fusion weight value, the target image representation vector and the target text representation vector.
[0017] According to another aspect of the present disclosure, there is also provided an article aggregation device, comprising:
[0018] A data acquisition module, which acquires target image data and target text data corresponding to the target object;
[0019] An image generation module, inputting the target image data into an image encoder model to obtain a target image representation vector;
[0020] A text generation module, inputting the target text data into a text encoder model to obtain a target text representation vector;
[0021] A vector fusion module, which obtains a target fusion representation vector according to the target image representation vector and the target text representation vector;
[0022] The item determination module determines similar items corresponding to the target item according to the target fusion representation vector based on a density clustering algorithm.
[0023] According to another aspect of the present disclosure, an electronic device is also provided, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned item aggregation methods by executing the executable instructions.
[0024] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for aggregating items described in any one of the above is implemented.
[0025] The object aggregation method, device, electronic device and computer-readable storage medium provided by the embodiments of the present disclosure obtain target image data and target text data corresponding to the target object, input the target image data into the image encoder model to obtain the target image representation vector, input the target text data into the text encoder model to obtain the target text representation vector, obtain the target fusion representation vector according to the fusion weight value, the target image representation vector and the target text representation vector, determine the clustering cluster corresponding to the target fusion representation vector according to the target fusion representation vector and feature pool data based on the density clustering algorithm, determine the similar objects corresponding to the target object according to the clustering cluster, can efficiently determine the similar objects corresponding to the target object without relying on manual labeling, and has strong scalability.
[0026] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.
[0028] Figure 1 A flow chart of an item aggregation method in an embodiment of the present disclosure is shown;
[0029] Figure 2 A flow chart of a text encoder model training method according to an embodiment of the present disclosure is shown;
[0030] Figure 3 A flow chart of a method for training an image encoder model in an embodiment of the present disclosure is shown;
[0031] Figure 4 A schematic diagram of an article aggregation device according to an embodiment of the present disclosure is shown;
[0032] Figure 5 A flow chart of another method for aggregating items in an embodiment of the present disclosure is shown;
[0033] Figure 6 A schematic diagram showing the principle of self-supervised pre-training of object images in an embodiment of the present disclosure is shown;
[0034] Figure 7 A flow chart of a similar item aggregation method according to an embodiment of the present disclosure is shown;
[0035] Figure 8 A schematic diagram showing an exemplary system architecture that can be applied to the item aggregation method or item aggregation device of the embodiments of the present disclosure; and
[0036] Fig. 9 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0037] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0038] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.
[0039] The present exemplary implementation is described in detail below with reference to the accompanying drawings and embodiments.
[0040] First, an embodiment of the present disclosure provides an item aggregation method, which can be executed by any electronic device with computing and processing capabilities.
[0041] Figure 1 A flowchart of an item aggregation method in an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the method for aggregating items provided in the embodiment of the present disclosure includes the following steps:
[0042] S102, acquiring target image data and target text data corresponding to the target object.
[0043] In one embodiment, the target items include, but are not limited to: commodities, products, virtual currencies, etc. for trading.
[0044] In one embodiment, the target text data includes, but is not limited to: item title data, item type, item attribute, item label, item model, item color, item detail data, etc.
[0045] In one embodiment, the target image data includes but is not limited to: object picture data, object video data, etc.
[0046] S104, inputting the target image data into the image encoder model to obtain a target image representation vector.
[0047] In one embodiment, one or more negative sample items corresponding to the positive sample item are determined based on the positive sample item; an image loss function is determined based on the image data of the positive sample item and the negative sample item; and an image encoder model is trained based on the image loss function.
[0048] The target image representation vector is a visual representation vector.
[0049] In one embodiment, the image encoder model is a picture encoder model or a video encoder model.
[0050] In one embodiment, the picture encoder model includes but is not limited to: ResNet50, Vision-Transformer, etc.
[0051] S106, input the target text data into the text encoder model to obtain a target text representation vector.
[0052] In one embodiment, the text encoder model may be a transformer architecture, etc.
[0053] In one embodiment, one or more negative sample items corresponding to the positive sample item are determined based on the positive sample item; a text loss function is determined based on the positive sample item and the negative sample item; and a text encoder model is trained based on the text loss function.
[0054] The target text representation vector is a sentence vector.
[0055] S108, obtaining a target fusion representation vector according to the target image representation vector and the target text representation vector.
[0056] In one embodiment, a fusion weight value is obtained; a target fusion representation vector is obtained based on the fusion weight value, the target image representation vector and the target text representation vector; the fusion weight value can be determined by experimental verification or user needs, and the sum of the fusion weight values corresponding to the target image representation vector and the target text representation vector is 1.
[0057] S110, based on a density clustering algorithm, determining similar items corresponding to the target item according to the target fusion representation vector.
[0058] In one embodiment, based on a density clustering algorithm, a cluster corresponding to the target fusion representation vector is determined according to the target fusion representation vector and feature pool data; and similar items corresponding to the target item are determined according to the cluster cluster.
[0059] In one embodiment, a fusion representation vector of multiple items in an item library is obtained to obtain a feature pool, and clusters corresponding to each fusion representation vector in the feature pool are determined based on a density clustering algorithm, and a cluster mapping table is generated. The cluster mapping table includes but is not limited to: fusion representation vectors, clusters, item identifiers, etc.; a cluster is determined according to a target fusion representation vector, and other items included in the cluster are determined according to the cluster, that is, similar items.
[0060] In one embodiment, the size of the clustering clusters can be controlled by adjusting parameters such as the distance threshold of the neighborhood and the number of core samples.
[0061] In one embodiment, the density clustering algorithm includes but is not limited to: DBSCAN (Density-Based Spatial Clustering of Applications with Noise) or OPTICS (Ordering points to identify the clustering structure, an algorithm for identifying clustering structure by sorting sample points).
[0062] In the above embodiment, the image data and text description information of the object are taken into consideration at the same time, and a fused representation of the image and text is obtained through self-supervised training. In the fused feature space, a density-based clustering algorithm is used to obtain clusters of similar products. This method can efficiently determine similar items corresponding to the target item without relying on manual labeling, and has strong scalability.
[0063] Figure 2 A flow chart of a text encoder model training method according to an embodiment of the present disclosure is shown as follows: Figure 2 As shown, the text encoder model training method provided in the embodiment of the present disclosure includes the following steps:
[0064] S202: Determine one or more negative sample items corresponding to the positive sample item according to the positive sample item.
[0065] In one embodiment, in the item library, all items other than positive sample items are negative sample items.
[0066] S204, inputting the positive text data of the positive sample item into the text encoder model to obtain a plurality of positive text representation vectors.
[0067] In one embodiment, the positive text data of the positive sample items are input into the text encoder model multiple times to obtain multiple positive text representation vectors; it should be noted that the text encoder model has a random dropout operation, and the text encoder model will generate different multiple positive text representation vectors for the positive text data input multiple times.
[0068] S206, inputting the negative text data of the negative sample items into the text encoder model to obtain multiple negative text representation vectors.
[0069] It should be noted that the number of positive text representation vectors and negative text representation vectors should be consistent.
[0070] S208, determining a text loss function according to the positive text representation vector and the negative text representation vector.
[0071] In one embodiment, the text loss function may be a cross entropy loss or the like.
[0072] S210, training a text encoder model based on the text loss function.
[0073] In one embodiment, the training goal is that multiple positive text representation vectors should be similar and dissimilar to other negative text representation vectors in the same batch. The text encoder model can be trained using methods such as cosine similarity, Euclidean distance, and Manhattan distance.
[0074] In the above embodiment, the text encoder model is trained by self-supervised training to improve the accuracy of the text encoder model, thereby obtaining a fused representation of the image and text. In the fused feature space, a density-based clustering algorithm is used to obtain clusters of similar products. This method can efficiently determine similar items corresponding to the target item without relying on manual labeling, and has strong scalability.
[0075] Figure 3 A flow chart of a method for training an image encoder model in an embodiment of the present disclosure is shown as follows: Figure 3 As shown, the image encoder model training method provided in the embodiment of the present disclosure includes the following steps:
[0076] S302: Determine one or more negative sample items corresponding to the positive sample item according to the positive sample item.
[0077] S304, obtaining a plurality of positive image data of the positive sample objects based on a plurality of data enhancement methods.
[0078] In one embodiment, the data enhancement method includes but is not limited to at least one of the following: random cropping, horizontal or vertical flipping, random rotation, Gaussian blur, grayscale transformation, brightness transformation, contrast adjustment, saturation adjustment, etc.
[0079] S306, inputting a plurality of positive image data of the positive sample objects into the image encoder model to obtain a plurality of positive image representation vectors.
[0080] S308, obtaining multiple negative image data of the negative sample objects based on multiple data enhancement methods.
[0081] S310, inputting a plurality of negative image data of negative sample objects into an image encoder model to obtain a plurality of negative image representation vectors.
[0082] In one embodiment, the positive image data is the picture and image data corresponding to the positive sample object; when the positive image data is picture data, correspondingly, the negative image data is also picture data, and the image encoder model is a picture encoder model; when the positive image data is video data, correspondingly, the negative image data is also video data, and the image encoder model is a video encoder model.
[0083] S312, determining an image loss function according to the positive image representation vector and the negative image representation vector.
[0084] It should be noted that the positive image representation vector and the negative image representation vector are visual representation vectors; the positive image representation vectors attract each other, the negative image representation vectors attract each other, and the positive image representation vector and the negative image representation vector repel each other.
[0085] S314, training an image encoder model based on the image loss function.
[0086] The training goal is to reduce the distance between representations of the same object (after data augmentation) in the image representation space, while increasing the distance between representations of different objects.
[0087] In the above embodiment, the image encoder model is trained by self-supervised training to improve the accuracy of the image encoder model, thereby obtaining a fused representation of the image and text. In the fused feature space, a density-based clustering algorithm is used to obtain clusters of similar products. This method can efficiently determine similar items corresponding to the target item without relying on manual labeling, and has strong scalability.
[0088] Based on the same inventive concept, the present disclosure also provides an article aggregation device, such as the following embodiment. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.
[0089] Figure 4 A schematic diagram of an article aggregation device in an embodiment of the present disclosure is shown. Figure 4 As shown, the object aggregation device 4 includes: a data acquisition module 401, an image generation module 402, a text generation module 403, a vector fusion module 404, and an object determination module 405;
[0090] The data acquisition module 401 acquires target image data and target text data corresponding to the target object;
[0091] The image generation module 402 inputs the target image data into the image encoder model to obtain the target image representation vector;
[0092] The text generation module 403 inputs the target text data into the text encoder model to obtain the target text representation vector;
[0093] A vector fusion module 404 obtains a target fusion representation vector according to the target image representation vector and the target text representation vector;
[0094] The item determination module 405 determines similar items corresponding to the target item according to the target fusion representation vector based on a density clustering algorithm.
[0095] In one embodiment, the item determination module 405 is also used to: obtain the fusion representation vectors of multiple items in the item library to obtain a feature pool, determine the cluster clusters corresponding to each fusion representation vector in the feature pool based on a density clustering algorithm, and generate a cluster mapping table, the cluster mapping table includes but is not limited to: fusion representation vectors, cluster clusters, item identifiers, etc.; determine the cluster cluster according to the target fusion representation vector, and determine other items included in the cluster cluster according to the cluster cluster, that is, similar items.
[0096] In one embodiment, the item aggregation device 4 also includes a text training module, which determines one or more negative sample items corresponding to the positive sample item based on the positive sample item; inputs the positive text data of the positive sample item into the text encoder model to obtain multiple positive text representation vectors; inputs the negative text data of the negative sample item into the text encoder model to obtain multiple negative text representation vectors; determines a text loss function based on the positive text representation vectors and the negative text representation vectors; and trains the text encoder model based on the text loss function.
[0097] In one embodiment, the object aggregation device 4 also includes an image training module, which determines one or more negative sample objects corresponding to the positive sample object based on the positive sample object; obtains multiple positive image data of the positive sample object based on multiple data enhancement methods; inputs the multiple positive image data of the positive sample object into the image encoder model to obtain multiple positive image representation vectors; obtains multiple negative image data of the negative sample object based on multiple data enhancement methods; inputs the multiple negative image data of the negative sample object into the image encoder model to obtain multiple negative image representation vectors; determines the image loss function according to the positive image representation vectors and the negative image representation vectors; and trains the image encoder model based on the image loss function.
[0098] In the above embodiment, the image data and text description information of the object are taken into consideration at the same time, and a fused representation of the image and text is obtained through self-supervised training. In the fused feature space, a density-based clustering algorithm is used to obtain clusters of similar products. This method can efficiently determine similar items corresponding to the target item without relying on manual labeling, and has strong scalability.
[0099] Figure 5 FIG. 1 shows another flowchart of an item aggregation method in an embodiment of the present disclosure. The present disclosure embodiment is introduced by taking the target text data as title data and the image encoder model as a picture encoder as an example. Figure 5 As shown, the method for aggregating items provided in the embodiment of the present disclosure includes the following steps:
[0100] S502, training a text encoder model on the item title data in a self-supervised manner.
[0101] The training steps are as follows:
[0102] 1. Input the same item title (positive sample) into the pre-trained text encoder model twice. Due to the presence of random inactivation dropout operation in the model, the two forward processes of the text encoder model will generate two different positive text representation vectors, which can be represented by sentence embedding. These two positive text representation vectors can be used as "positive sample pairs".
[0103] In one embodiment, the pre-trained text encoder model may be a transformer architecture.
[0104] It should be noted that the same article title can be input into the pre-trained text encoder model multiple times to obtain multiple positive text representation vectors. The embodiment of the present disclosure takes two times as an example for introduction.
[0105] 2. Treat all other article title samples in the same batch as negative samples corresponding to the current positive sample, and operate in the same way as above, that is, input all other negative samples in the same batch into the pre-trained text encoder model twice to generate two different negative text representation vectors, and use these two negative text representation vectors as "negative sample pairs" to train the text encoder model using the following text loss function:
[0106]
[0107] Wherein, sim is a vector similarity calculation function (such as cosine similarity, etc.);
[0108] t1 is the temperature parameter, which has a value between 0 and 1 and can be adjusted;
[0109] N1 is the batch size;
[0110] represents the different text representation vectors generated by the forward process of the 2nd text encoder model in the above step for the i-th sample in the current batch; and represents the text representation vector generated by the jth sample in the current batch. There are N samples in the current batch, so the denominator contains a summation operation;
[0111] The meaning of this formula is that the text representation vector obtained by forwarding the text encoder model twice for the i-th sample in a batch should be similar, but not similar to the text representation vectors of the remaining N-1 samples in the same batch. Therefore, the text loss function here uses the cross entropy loss often used in classification functions.
[0112] It should be noted that this solution does not restrict the selection of sim function, t1, and N1.
[0113] 3. Train the text encoder model using the text loss function until convergence.
[0114] S504, training the image encoder model on the object image data in a self-supervised manner. The training steps are as follows:
[0115] Figure 6 A schematic diagram of the principle of self-supervised pre-training of object images in an embodiment of the present disclosure is shown. Figure 6 As shown:
[0116] 1. The same positive sample item image is processed by different data enhancement methods to obtain two different images, namely the first image and the second image. These two different images are respectively passed through the image encoder model;
[0117] In one embodiment, the data enhancement method includes: random cropping, horizontal or vertical flipping, random rotation, Gaussian blur, grayscale transformation, brightness transformation, contrast adjustment, saturation adjustment, etc.
[0118] In one embodiment, commonly used image encoder models include ResNet50, Vision-Transformer, etc.
[0119] 2. The first image and the second image obtained through the above steps are regarded as a "positive sample pair". The positive image representation vectors obtained by the image encoder model of these two images are respectively recorded as the first positive image representation vector z i and the second positive image representation vector z j , the first positive image characterization vector and the second positive image characterization vector attract each other.
[0120] Other pictures in the same batch are determined as negative sample item pictures corresponding to the positive sample item pictures, and the negative sample item pictures are processed by different data enhancement methods to obtain two different pictures, that is, the third picture and the fourth picture are regarded as "negative sample pairs", and the two different pictures are respectively passed through the picture encoder model to obtain negative image representation vectors, that is, the first negative image representation vector and the second negative image representation vector, the first negative image representation vector and the second negative image representation vector attract each other; but the positive image representation vector and the negative image representation vector repel each other.
[0121] The image encoder model is trained using the following image loss function:
[0122]
[0123] Among them, t2 is the temperature parameter;
[0124] N2 is the batch size;
[0125] 1[k≠i] is the indicator function, which means that the sample is not compared with itself;
[0126] zi,z j represents the different image representation vectors corresponding to the i-th sample in the current batch; and z in the denominator k Represents the image representation vector generated by the j-th sample in the current batch;
[0127] The training goal is to reduce the distance between representations of the same object (after data augmentation) in the image representation space, while increasing the distance between representations of different objects.
[0128] In one embodiment, the sizes of t1 and t2, N1 and N2 may be different, and the comparison is not limited.
[0129] The image encoder model is trained using this image loss function until convergence.
[0130] It should be noted that the above formula (2) can also be applied to train video encoders, etc.
[0131] S506, obtaining a fused vector representation of the object based on the image encoder model and the text encoder model;
[0132] Figure 7 A flow chart of a similar item aggregation method according to an embodiment of the present disclosure is shown as follows: Figure 7 As shown, for each item in the item library, its item picture and item title are input into the image encoder model and the text encoder model respectively to obtain its image representation vector E I and the text representation vector E T , use the following calculation method to calculate the item fusion representation vector E M :
[0133] E M =αE I +βE T
[0134] Among them, α and β are fusion weight parameters respectively. The weight parameters are hyperparameters that can adjust the weights of the two modal information. The optimal value can be determined through experimental verification or user needs. It only requires that the sum of α and β parameters is 1.
[0135] S508, after the above steps, the fusion vector representation of each item is obtained; each item in the item library is processed in the same way to obtain a feature pool containing all item information. A density-based clustering algorithm (such as DBSCAN, OPTICS, etc.) is used on this feature pool. The density clustering algorithm examines the connectivity between samples from the perspective of sample density, and continuously expands clusters based on the connectable samples to obtain the final clustering result.
[0136] It should be noted that this solution requires that the clustering algorithm be a density-based clustering algorithm, rather than a partitioning clustering algorithm (such as k-means, etc.). The advantages of density clustering are: 1. There is no need to specify the number of clusters in advance; 2. It can process non-convex data, and the clusters can be of any shape.
[0137] S510, each item in the item library can be assigned to a cluster, and items in the same cluster are similar items, that is, similar items are determined according to the clusters to which the items are located.
[0138] In one embodiment, the size of the clustering clusters can be controlled by adjusting parameters such as the distance threshold of the neighborhood and the number of core samples.
[0139] The above embodiment can be applied to e-commerce business scenarios, and the fusion representation of pictures and texts is obtained by self-supervised training. In the fused feature space, a density-based clustering algorithm is used to obtain similar product clusters. This process does not rely on manual labeling and has strong scalability.
[0140] Figure 8 A schematic diagram showing an exemplary system architecture that can be applied to the item aggregation method or item aggregation device according to embodiments of the present disclosure.
[0141] like Figure 8 As shown, the system architecture 800 may include terminal devices 801 , 802 , 803 , a network 804 and a server 805 .
[0142] The network 804 is a medium for providing a communication link between the terminal devices 801 , 802 , 803 and the server 805 , and may be a wired network or a wireless network.
[0143] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged through the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.
[0144] The terminal devices 801 , 802 , and 803 may be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.
[0145] Optionally, the client of the application installed in different terminal devices 801, 802, 803 is the same, or is a client of the same type of application based on different operating systems. Based on different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile client, a PC client, etc.
[0146] The server 805 may be a server that provides various services, such as a background management server that provides support for the devices operated by the user using the terminal devices 801, 802, and 803. The background management server may analyze and process the received request data, and feed back the processing results to the terminal device, such as obtaining the target image data and target text data corresponding to the target object; training the image encoder model, the text encoder model, etc.
[0147] Optionally, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.
[0148] Those skilled in the art will know that Figure 8 The number of terminal devices, networks and servers in the embodiment is only for illustration, and any number of terminal devices, networks and servers may be provided according to actual needs, and the embodiments of the present disclosure do not limit this.
[0149] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".
[0150] Refer to the following Fig. 9 The electronic device 900 according to this embodiment of the present disclosure is described. Fig. 9 The electronic device 900 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0151] like Fig. 9 As shown, the electronic device 900 is in the form of a general computing device. The components of the electronic device 900 may include but are not limited to: at least one processing unit 910, at least one storage unit 920, and a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910).
[0152] The storage unit stores program codes, which can be executed by the processing unit 910, so that the processing unit 910 executes the steps described in the above “Exemplary Method” section of this specification according to various exemplary embodiments of the present disclosure.
[0153] For example, the processing unit 910 can execute the following steps of the above method embodiment: obtain target image data and target text data corresponding to the target item, input the target image data into the image encoder model to obtain a target image representation vector, input the target text data into the text encoder model to obtain a target text representation vector, obtain a target fusion representation vector based on a fusion weight value, a target image representation vector and a target text representation vector, determine a clustering cluster corresponding to the target fusion representation vector based on the target fusion representation vector and feature pool data based on a density clustering algorithm, and determine similar items corresponding to the target item based on the clustering cluster.
[0154] For example, the processing unit 910 can execute the following steps of the above method embodiment: determine one or more negative sample items corresponding to the positive sample item based on the positive sample item; obtain multiple positive image data of the positive sample item based on multiple data enhancement methods; input the multiple positive image data of the positive sample item into the image encoder model to obtain multiple positive image representation vectors; obtain multiple negative image data of the negative sample item based on multiple data enhancement methods; input the multiple negative image data of the negative sample item into the image encoder model to obtain multiple negative image representation vectors; determine the image loss function based on the positive image representation vectors and the negative image representation vectors; and train the image encoder model based on the image loss function.
[0155] The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 9201 and / or a cache storage unit 9202 , and may further include a read-only storage unit (ROM) 9203 .
[0156] The storage unit 920 may also include a program / utility 9204 having a set (at least one) of program modules 9205, such program modules 9205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0157] Bus 930 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0158] The electronic device 900 may also communicate with one or more external devices 940 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 900, and / or may communicate with any device that enables the electronic device 900 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 950. Furthermore, the electronic device 900 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 960. As shown, the network adapter 960 communicates with other modules of the electronic device 900 via a bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.
[0159] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.
[0160] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the above method of the present disclosure is stored thereon. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.
[0161] For example, when the program product in the embodiment of the present disclosure is executed by a processor, the method implements the following steps: obtaining target image data and target text data corresponding to the target item, inputting the target image data into the image encoder model to obtain a target image representation vector, inputting the target text data into the text encoder model to obtain a target text representation vector, obtaining a target fusion representation vector based on a fusion weight value, the target image representation vector and the target text representation vector, determining a clustering cluster corresponding to the target fusion representation vector based on the target fusion representation vector and feature pool data based on a density clustering algorithm, and determining similar items corresponding to the target item based on the clustering cluster.
[0162] For example, when the program product in the embodiments of the present disclosure is executed by a processor, the following steps are implemented: determining one or more negative sample items corresponding to the positive sample item based on the positive sample item; inputting the positive text data of the positive sample item into the text encoder model to obtain multiple positive text representation vectors; inputting the negative text data of the negative sample item into the text encoder model to obtain multiple negative text representation vectors; determining a text loss function based on the positive text representation vectors and the negative text representation vectors; and training the text encoder model based on the text loss function.
[0163] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0164] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0165] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.
[0166] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).
[0167] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.
[0168] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.
[0169] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.
[0170] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.
Claims
1. A method for aggregating items, characterized in that: include: Obtain target image data and target text data corresponding to the target object; Inputting the target image data into an image encoder model to obtain a target image representation vector; Inputting the target text data into a text encoder model to obtain a target text representation vector; Obtaining a target fusion representation vector according to the target image representation vector and the target text representation vector; Based on a density clustering algorithm, similar items corresponding to the target item are determined according to the target fusion representation vector.
2. The method for aggregating items according to claim 1, characterized in that: The method of determining similar items corresponding to the target item according to the target fusion representation vector based on the density clustering algorithm includes: Based on the density clustering algorithm, determine the cluster cluster corresponding to the target fusion representation vector according to the target fusion representation vector and feature pool data; The similar items corresponding to the target item are determined according to the clusters.
3. The method for aggregating articles according to claim 1 or 2, characterized in that: The density clustering algorithm includes: a density-based clustering method with noise or an algorithm for identifying clustering structure by sorting sample points.
4. The method for aggregating items according to claim 1, characterized in that: Also includes: Determine one or more negative sample items corresponding to the positive sample item according to the positive sample item; Determine a text loss function according to the text data of the positive sample item and the negative sample item; The text encoder model is trained based on the text loss function.
5. The method for aggregating items according to claim 4, characterized in that: Determining the text loss function according to the positive sample items and the negative sample items includes: Inputting the positive text data of the positive sample item into the text encoder model to obtain a plurality of positive text representation vectors; Inputting the negative text data of the negative sample items into the text encoder model to obtain a plurality of negative text representation vectors; The text loss function is determined according to the positive text representation vector and the negative text representation vector.
6. The method for aggregating items according to claim 1, characterized in that: Also includes: Determine one or more negative sample items corresponding to the positive sample item according to the positive sample item; Determining an image loss function according to the image data of the positive sample object and the negative sample object; The image encoder model is trained based on the image loss function.
7. The method for aggregating items according to claim 6, characterized in that: The determining of the image loss function according to the positive sample object and the negative sample object comprises: Inputting a plurality of positive image data of the positive sample object into the image encoder model to obtain a plurality of positive image representation vectors; Inputting a plurality of negative image data of the negative sample objects into the image encoder model to obtain a plurality of negative image representation vectors; The image loss function is determined according to the positive image characterization vector and the negative image characterization vector.
8. The method for aggregating items according to claim 7, characterized in that: Also includes: The positive sample objects are subjected to multiple data enhancement methods to obtain the multiple positive image data; The negative sample objects are subjected to multiple data enhancement methods to obtain the multiple negative image data.
9. The method for aggregating objects according to claim 8, characterized in that: The data enhancement method includes at least one of the following: random cropping, horizontal or vertical flipping, random rotation, Gaussian blur, grayscale transformation, brightness transformation, contrast adjustment, and saturation adjustment.
10. The method for aggregating articles according to claim 1 or 6, characterized in that: The image encoder model is a picture encoder model or a video encoder model.
11. The method for aggregating items according to claim 1, characterized in that: The obtaining of the target fusion representation vector according to the target image representation vector and the target text representation vector comprises: Get the fusion weight value; The target fusion representation vector is obtained according to the fusion weight value, the target image representation vector and the target text representation vector.
12. An article aggregation device, characterized in that: include: A data acquisition module, which acquires target image data and target text data corresponding to the target object; An image generation module, inputting the target image data into an image encoder model to obtain a target image representation vector; A text generation module, inputting the target text data into a text encoder model to obtain a target text representation vector; A vector fusion module, which obtains a target fusion representation vector according to the target image representation vector and the target text representation vector; The item determination module determines similar items corresponding to the target item according to the target fusion representation vector based on a density clustering algorithm.
13. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the item aggregation method described in any one of claims 1 to 11 by executing the executable instructions.
14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the object aggregation method described in any one of claims 1 to 11 is implemented.