Data processing method and device, medium and program product

Abstract

The invention relates to the field of computer data processing, in particular to a user generated content-based data processing method, which comprises the following steps of: identifying, namely identifying named entities in notes, and acquiring category words of the named entities; an extraction step: determining a context corresponding to the category word in the note based on the category word; a recall step: performing first recall and / or second recall on predetermined commodity categories based on the category words to select candidate items; and a sorting step: calculating the confidence of the candidate items based on the context, and determining a target commodity category associated with the user generated content from the candidate items based on the confidence. The invention also relates to a user-generated content-based data processing device, a medium and a program product. According to the method, the recall robustness can be improved, and the cost of manually annotating data can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer data processing, and in particular to a data processing method, device, medium and program product based on user-generated content. Background Art

[0002] Users will post a lot of notes on the Internet, namely user-generated content (UGC), to express their opinions or interests. Most commonly, users will share their product experience with other users. In these notes, it will be mentioned what product the main content of the current note is about, and the main category information of the note will appear. At this time, it is necessary to construct a mapping relationship from the note information to the product category to classify the current note and then profile the user who posted the note. For example, if the current note is describing a pair of women's trousers, it can be classified into the product category of clothing, shoes and bags, women's boutique, bottoms, and pants.

[0003] At present, the entity linking technology that constructs a mapping relationship from note information to product categories mainly includes recall and sorting. The existing recall is mainly based on a rule-based approach, which has limitations and lacks robustness. It can only be recalled in a fixed way. The sorting stage requires manual labeling of data and training of sorting models, so the labor cost is very high. In addition, for new fields, new labeled data and training models are required, and the transferability is very poor. How to increase the robustness of recall and the transferability of sorting is a difficult problem. Summary of the invention

[0004] The purpose of the present invention is to provide a data processing method, device, medium and program product based on user-generated content. By using vector recall based on minimum semantic units while performing at least one level of recall, the interpretability of the recall can be increased, and the robustness of the recall can be increased. The dual-form sorting method of the user-generated content side and the product category side can effectively reduce the size of the training set.

[0005] An embodiment of the present invention discloses a data processing method based on user-generated content, the method comprising:

[0006] An identification step of identifying named entities in the user-generated content and obtaining category words of the named entities;

[0007] An extraction step of determining a context corresponding to the category word in the user-generated content based on the category word;

[0008] A recall step, based on the category words, performing a first recall and / or a second recall on the predetermined product category to select candidate items;

[0009] The sorting step is to calculate the confidence of the candidate items based on the context, and determine the target product category associated with the user-generated content from the candidate items based on the confidence.

[0010] Optionally, the context corresponding to the category word in the extraction step includes the sentence, title, topic and any combination thereof in which the category word is located.

[0011] Optionally, the first recall in the recall step is at least one level of recall based on predetermined rules.

[0012] Optionally, the at least one level recall includes:

[0013] First-level recall, the predetermined rule of the first-level recall is that the category word is completely consistent with the leaf node of the commodity category;

[0014] Secondary recall, the predetermined rule of the secondary recall is that the segmentation result of the category word is completely included in the segmentation result of the leaf node of the commodity category;

[0015] And, a third-level recall, the predetermined rule of the third-level recall is that more than 50% of the word segmentation results of the category words are included in the word segmentation results of the leaf nodes of the commodity category.

[0016] Optionally, the second recall in the recall step is a token-based vector recall, wherein a token represents the smallest semantic unit of the category word.

[0017] Optionally, calculating the confidence of the candidate item based on the context includes calculating the confidence of the candidate item based on a twin-tower model, the twin-tower model including a user-generated content side and a product category side, the input of the user-generated content side includes the context, and the input of the product category side is text in the product database.

[0018] Optionally, the input on the user-generated content side further includes pictures in the user-generated content, and the input on the product category side further includes pictures in the product database.

[0019] Optionally, calculating the confidence of the candidate item based on the context further includes calculating the confidence of the candidate item based on the context and discrete features of the candidate item.

[0020] An embodiment of the present invention discloses an electronic device, characterized in that the device includes a processor and a memory storing computer-executable instructions, and the processor is configured to execute the instructions to implement the above-mentioned data processing method based on user-generated content.

[0021] An embodiment of the present invention discloses a computer-readable storage medium, characterized in that at least one computer instruction is stored in the computer-readable storage medium, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned data processing method based on user-generated content.

[0022] An embodiment of the present invention discloses a computer program product, which includes computer instructions. When the computer instructions are executed, the above-mentioned data processing method based on user-generated content is implemented.

[0023] The present invention provides a data processing method, device, medium and program product based on user-generated content, which adopts at least one level of rule recall combined with vector recall to increase the interpretability and robustness of recall, and adopts a double-tower model with sorting on the user-generated content side and the product category side to sort the recall results.

[0024] Compared with the prior art, the main differences and effects of the embodiments of the present invention are: while ensuring rule recall, increasing interpretability, and facilitating manual adjustment and intervention, the robustness of recall can be increased by using a vector recall method; a dual form of sorting on the user-generated content side and the product category side is adopted to extract feature information to complete the link to the product category, and the required training set size is small, the manual annotation cost is low, and the transferability is strong. The product category side can allow offline feature extraction, further ensuring the timeliness of the model online. The input of discrete features is increased to further improve the model effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a scenario schematic diagram of a data processing method based on user-generated content according to an embodiment of the present invention;

[0026] Figure 2 is a flow chart of a data processing method based on user-generated content according to an embodiment of the present invention;

[0027] Figure 3 is a structural schematic diagram of a sorting method according to an embodiment of the present invention;

[0028] Figure 4 is a hardware structure block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described below in conjunction with specific embodiments and accompanying drawings. It will be appreciated that the specific embodiments described herein are intended only to explain the present invention, rather than to limit the present invention. In addition, for ease of description, only some, but not all, structures or processes related to the present invention are shown in the accompanying drawings. It should be noted that in this specification, similar reference numerals and letters represent similar items in the following drawings.

[0030] It should be understood that although the terms "first", "second", etc. may be used herein to describe various features, these features should not be limited by these terms. These terms are used only to distinguish and should not be understood as indicating or implying relative importance. For example, a first feature may be referred to as a second feature, and similarly a second feature may be referred to as a first feature without departing from the scope of the exemplary embodiments.

[0031] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0032] Figure 1 4 is a schematic diagram of a scenario of a data processing method based on user-generated content according to an embodiment of the present invention.

[0033] like Figure 1 As shown, the application scenario of the present invention includes a client 100 and a server 200, and the client 100 and the server 200 achieve communication through a wired network or a wireless network connection.

[0034] It is worth noting that those skilled in the art will understand that there can be one or more clients 100, and the number of clients 100 should not be used as a limiting condition of the present invention. The number of clients 100 is based on the actual needs in the application scenario. For example, in a user-generated content recommendation system, the data source comes from different clients 100, and at least one client 100 sends data to the recommendation system. The client 100 can be a computer device such as a smart phone, a tablet computer, a user-generated content laptop, a desktop computer, etc., which has computing power, input and output functions, and network connection functions. The server 200 can be a single server device, or it can be a computer device such as a server cluster, a cloud server, etc. that provides game application services to the outside.

[0035] The current recall has great limitations and is not robust enough. Sorting requires manual data labeling and training of sorting models, which consumes a lot of labor costs. When facing new fields, new labeled data and training models are required. Therefore, how to increase the robustness of recall and the transferability of sorting is a difficult problem.

[0036] In response to the above problems, the present invention provides a data processing method, device, medium and program product based on user-generated content, which can increase the interpretability and robustness of recall, and can also effectively reduce the size of the training set and increase the transferability of the sorting.

[0037] Figure 2 is a flow chart of a data processing method based on user-generated content according to an embodiment of the present invention.

[0038] like Figure 2 As shown, the data processing method based on user-generated content provided by an embodiment of the present invention requires Figure 1 The method is implemented by the cooperation of the client 100 and the server 200, and includes:

[0039] Identification step S100: identifying named entities in user-generated content and obtaining category words of the named entities.

[0040] As an implementation method, Stanford CoreNLP or pyltp is used to identify named entities in user-generated content.

[0041] Stanford CoreNLP is a natural language processing tool whose main functions include word segmentation, part-of-speech tagging, named entity recognition, syntactic structure analysis and dependency analysis.

[0042] As a tool for natural language processing, pyltp is a Python encapsulation of LTP, which provides functions of word segmentation, part-of-speech tagging, named entity recognition, dependency syntax analysis, and semantic role labeling.

[0043] It is worth noting that the purpose of using a tool to identify named entities of user-generated content is to extract category words, and it is assumed that those skilled in the art can extract category words. The identification tool should not be used as a limitation of the present invention.

[0044] Named entity recognition refers to the identification of entities with specific meanings in text, mainly including names of people, places, proper nouns, organization names, etc. Generally speaking, the task of named entity recognition is to identify three major categories of named entities in the text to be processed, such as entity category, time category and number category, as well as seven subcategories, such as names of people, organization names, place names, time, date, currency and percentage. Named entity recognition aims to identify entities in a string of text and mark the type it refers to, such as names of people, places, etc. Specifically, given an input sentence, named entity recognition requires entity annotation for each token in the sentence, such as names of people, organizations / institutions, place names, dates, etc. The token here is the smallest semantic unit.

[0045] The recognition result of the named entity is used as a candidate word, and special symbols are removed, and uppercase English and traditional Chinese characters are converted and other preprocessing is performed. In an example, a user publishes a content, and the user-generated content includes text and pictures, where the text content mainly introduces women's lipstick, and the picture content is women's makeup. The server 200 makes entity annotations for each token in the user-generated content, identifies the named entities in the user-generated content, and obtains the category words "lipstick" and "women's makeup".

[0046] In related technologies, the data sources for recall are relatively single, and most of them only consider text without considering the OCR information in the image, which is quite limited.

[0047] The named entities including product attributes in the user-generated content can be determined based on the text in the user-generated content, such as text and text in pictures. The text in the picture can be obtained through OCR technology. For example, in the picture, the image data can be divided first, the text information therein can be identified through OCR, or the corresponding text information can be obtained based on the feature vector of the image itself. In some examples, when the user-generated content includes a video, the video frame cutting method can be used to extract text information from each frame or certain specific frames of the picture.

[0048] Extraction step S200: determining the context corresponding to the category word in the user-generated content based on the category word.

[0049] As an implementation method, the context corresponding to the category word includes the sentence, title, topic and any combination thereof in which the category word is located. The picture is an image contained in the user-generated content itself. If the user-generated content includes a video, the key frame of the video is captured as the OCR recognition of the picture to extract the text information.

[0050] Determining the context corresponding to the category words in the user-generated content based on the category words is to extract the content related to the description of the category words.

[0051] In one example, based on the identified category words "lipstick" and "trousers", sentences containing "lipstick" in user-generated content are selected, such as the text information "bean paste-colored lipstick is the preferred color for women in spring" and a picture containing "women's makeup" information. The text information and the picture constitute the context of the current category words "lipstick" and "women's makeup".

[0052] Recall step S300: Based on the category words, a first recall and / or a second recall is performed on the predetermined commodity category to select candidate items. As an implementation method, the first recall is at least one level of recall based on a predetermined rule.

[0053] As another implementation, the second recall is a token-based vector recall.

[0054] It is worth noting that at least the first-level recall and the token-based vector recall can be performed simultaneously or according to priority, and finally all recall candidates of the first recall and the second recall are screened.

[0055] It can ensure the recall of rules while increasing the robustness of recall, and the correct recall literally has no similar correct candidates.

[0056] At least a level 1 recall includes level 1 recall, level 2 recall, and level 3 recall.

[0057] The combination logic of at least one level of recall is executed according to the priority, and finally a predetermined number of candidates are selected.

[0058] In one example, the system needs to recall 20 candidate items, and the priority arrangement is set to execute in the order of first-level recall, second-level recall, and third-level recall. The result of the first-level recall is to recall 10 candidate items, and the recall result does not meet the quantity requirement. Then the second-level recall is performed, and the result of the second-level recall is to recall 5 candidate items, and the recall result still does not meet the quantity requirement. It is necessary to continue to perform the third-level recall, and the result of the third-level recall is to recall 5 candidate items, and the multi-level recall task is completed.

[0059] In another example, the system needs to recall 20 candidate items, and the priority arrangement is set to execute in the order of first-level recall, second-level recall, and third-level recall. The result of the first-level recall is to recall 10 candidate items, and the recall result does not meet the quantity requirement. Then the second-level recall is performed, and the result of the second-level recall is to recall 10 candidate items, and the recall result meets the quantity requirement. There is no need to continue the third-level recall, and the multi-level recall task is completed.

[0060] In another example, the system needs to recall 20 candidate items, and the priority arrangement is set to execute in the order of first-level recall, second-level recall, and third-level recall. The result of the first-level recall is to recall 20 candidate items. The recall result meets the recall quantity requirement, and there is no need to continue to execute the second-level recall and the third-level recall. The multi-level recall task is completed.

[0061] The first recall is at least one level recall based on predetermined rules. The predetermined rules include the predetermined rules for the first level recall, the predetermined rules for the second level recall, and the predetermined rules for the third level recall, specifically including:

[0062] Level 1 recall: The predetermined rule for level 1 recall is that the category word is completely consistent with the leaf node of the product category.

[0063] First-level recall is precise recall, and it is a necessary condition that the category word is completely consistent with the leaf node in the product category.

[0064] Product categories may be multi-level. The bottom-level category nodes are called leaf nodes, and the leaf nodes of categories are called categories. Each recommended product must be placed in the leaf node of the category, that is, each recommended product must belong to a certain category. Recommended products are hung under the category.

[0065] In one example, the current category word is "shorts", and the product category is "Apparel, Shoes and Bags - Children's Clothing Selection - Children's Pants - Shorts", among which "shorts" belongs to the leaf node and can therefore be accurately recalled.

[0066] In another example, the current category word is "shorts" and the product category is "Apparel, Shoes and Bags - Children's Clothing Selection - Children's Pants - Trousers", among which "trousers" is a leaf node. The current category word "shorts" is not completely consistent with the leaf node "trousers" in the product category, and therefore, cannot be recalled.

[0067] Second-level recall: The predetermined rule for the second-level recall is that the segmentation results of the category words are completely included in the segmentation results of the leaf nodes of the product category.

[0068] The category list (list, also known as a linked list, is a data structure in which nodes are connected by fingers. Traversing the list means traversing all the nodes) is just one of the leaf nodes in the product category.

[0069] In one example, the current category word is "shorts", and the product category is "Apparel, Shoes and Bags - Pants Selection - Children's Pants - Children's Shorts", among which "Children's Shorts" belongs to the leaf node, "shorts" belongs to the word segmentation result, and the leaf node "Children's Shorts" contains the current category word "shorts", so it can be recalled by full token matching.

[0070] In one example, the current category word is "refrigerator", and the product category is "household appliances-kitchen appliances-kitchen refrigerators", where "kitchen refrigerator" belongs to the leaf node, "refrigerator" belongs to the word segmentation result, and the leaf node "kitchen refrigerator" contains the current category word "refrigerator", so it can be recalled by full token matching.

[0071] In another example, the current category word is "children's shorts" and the product category is "apparel, shoes and bags - selected pants - shorts". Among them, "shorts" belongs to the leaf node. The leaf node "shorts" does not contain the word segmentation result "children" in the current category word. Therefore, it cannot be recalled by full token matching.

[0072] The third-level recall has a predetermined rule that more than 50% of the segmentation results of the category words are included in the segmentation results of the leaf nodes of the product category.

[0073] Third-level recall is fuzzy recall, similar to synonym recall, or relational recall of hyponyms and hyponyms.

[0074] In one example, the current category word is "shorts" and the product category is "apparel, shoes and bags - children's clothing selection - children's trousers". Among them, "trousers" belongs to the word segmentation result of the leaf node and can be fuzzy recalled.

[0075] In another example, the current category word is "socks" and the product category is "Apparel, Shoes and Bags - Children's Clothing Selection - Children's Pants - Trousers", among which "trousers" is a leaf node. The current category word "socks" is completely inconsistent with the leaf node "trousers" in the product category. Therefore, it cannot be fuzzy recalled.

[0076] As an implementation method, token-based vector recall can adopt an embedding method to vectorize information. During recall, for each input category word, its embedding is calculated, and then the 20 clusters closest to it are recalled, that is, the 20 closest clusters are selected as recall candidates. Turning these 20 into recall candidates based on embedding under the category can effectively increase the robustness of recall.

[0077] Sorting step S400: calculating the confidence of the candidate items based on the context, and determining the target product category associated with the user-generated content from the candidate items based on the confidence.

[0078] In some embodiments, the confidence is calculated using the structure of a twin tower model. The twin towers include a twin tower model including a user-generated content side and a commodity category side. The input of the user-generated content side includes context, the input of the commodity category side is text in a commodity database, the input of the user-generated content side also includes pictures in the user-generated content, and the input of the commodity category side also includes pictures in the commodity database. The twin tower model as a whole includes an input layer, a representation layer, and a matching layer. In the input layer, the input of the user-generated content side includes text context and picture information extracted from the user-generated content, and the input of the commodity side is the text information of the commodity in the commodity database and multiple pictures under this category. The context here includes the context corresponding to the named entities in the text of the user-generated content, and may also include the context of the named entities in the text obtained in the picture by OCR (optical character recognition) technology.

[0079] In some other embodiments, the input layer may also include pictures. For example, the input on the user content generation side may also include pictures in the user generated content and / or screenshots in the video. Pictures are images contained in the user generated content itself. If the user generated content is in the form of a video, a key frame may be cut out every five seconds in the video and the key frame may be used as picture information. The input on the product side may include multiple pictures under the product category.

[0080] The representation layer in the dual-tower model can convert the input into a vector, and the matching layer can calculate the similarity of the vectors on the user-generated content side and the product side.

[0081] The data on the commodity side in the twin tower model can be calculated offline without affecting the real-time performance online. The picture is the image contained in the user-generated content itself. If the user-generated content is in the form of a video, a key frame can be cut out every five seconds in the video and the key frame can be used as the picture information. In addition, the input on the user content generation side can also be the text in the picture, such as the text converted from the picture according to the OCR (optical character recognition) technology.

[0082] In some embodiments, calculating the confidence of the candidate based on the context includes calculating the confidence of the candidate based on the context and discrete features of the candidate. Discrete features can be added to the matching layer of the twin tower model to increase the robustness of the model.

[0083] Use the context of the extracted category words to rank all candidate product categories.

[0084] In one example, the context of the extracted category words is used as text input, such as "exquisite women's socks", and is sorted with all candidate product categories "long socks, men's socks, boat socks, short socks".

[0085] The double-tower model structure is adopted. The left side is the text context and image information extracted from the user-generated content, and the right side is the text information of the product category and multiple images under this category. The data on the right side can be calculated offline without affecting the real-time performance online. The image is the image contained in the user-generated content itself. If the user-generated content is in the form of a video, the key frame of the video is captured.

[0086] Discrete features are used as intermediate inputs for confidence scoring. Discrete features include word overlap and popularity of product categories. For example, word overlap is the percentage of text information of a product category that appears in the context of the category; popularity is the number of times a product category appears in the training set.

[0087] Finally, the category word recommendation and the label and confidence of the candidate product category are output.

[0088] Figure 3 It is a structural schematic diagram of a sorting method according to an embodiment of the present invention.

[0089] like Figure 3 As shown in FIG. 1 , a dual-tower model is used to perform the sorting operation. The feature codes input from the user-generated content side (left side) and the product category side (right side) are concatenated into vectors, and then the similarity is obtained by calculating the cosine distance between the two vectors.

[0090] On the left side of the twin tower model, two BERT models are used to encode the context of named entities in user-generated content and OCR text in pictures in user-generated content, respectively. In addition, a ResNet model is used to encode the features of pictures in user-generated content. The vectors obtained from the three paths are concatenated, and the self-attention mechanism is used to consider the correlation between different vectors. When the user-generated content includes videos, key frames can be cut out of the video at each time interval, such as 5 seconds, and the key frames can be used as input pictures.

[0091] On the right side of the twin tower model, the BERT model is used to feature encode the text of the product category, and the ResNet model is used to feature encode multiple product images under this product category. The obtained vectors are cascaded, and the self-attention mechanism is used to consider the correlation between different vectors. The number of product images can be set by yourself, such as 10, 20 or 30.

[0092] As an implementation method, the confidence of the candidate items can be calculated based on discrete features. Discrete features are used as intermediate inputs for confidence scoring. For example, the degree of literal overlap can be calculated by measuring how much text information of a product category appears in the context information. Or the popularity of the current product category can be increased, that is, information such as product inventory, sales volume, and merchants under the current product category. The added discrete features can ensure the ability for manual intervention while increasing the robustness of the link.

[0093] exist Figure 3 In the twin tower model, the user-generated content side information and the product category side information are input, plus a little discrete feature, and finally the label label and confidence score score are output. The label indicates whether the current user-generated content is linked to the product 0 or 1, and the confidence indicates the probability of connecting to a certain product category in the product library. Under normal circumstances, once the target responds, the label is marked as 1. For example, according to the input on the left and right sides, the label between the user-generated content and product category 1 is 1, and the confidence is 0.9, while the similarity between the user-generated content and product category 2 meets a certain threshold, then the label is 1, but the confidence is 0.3. In this case, the user-generated content is linked to product category 1. If the label between the user-generated content and the product category is 0 according to the twin tower model, select another product category in the candidate and repeat the above input, comparison, label and confidence steps according to the twin tower model.

[0094] Figure 4 is a hardware structure block diagram of an electronic device according to an embodiment of the present invention.

[0095] like Figure 4As shown, the electronic device 400 may include one or more processors 402, a system control logic 408 connected to at least one of the processors 402, a system memory 405 connected to the system control logic 408, a non-volatile memory (NVM) 406 connected to the system control logic 408, and a network interface 410 connected to the system control logic 408.

[0096] The processor 402 may include one or more single-core or multi-core processors. The processor 402 may include any combination of general-purpose processors and special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In an embodiment of the present invention, the processor 402 may be configured to execute the following operations: Figure 2 The data processing method based on user-generated content is shown.

[0097] In some embodiments, system control logic 408 may include any suitable interface controller to provide any suitable interface to at least one of processors 402 and / or any suitable device or component in communication with system control logic 408 .

[0098] In some embodiments, the system control logic 408 may include one or more memory controllers to provide an interface to the system memory 405. The system memory 405 may be used to load and store data and / or instructions. In some embodiments, the system memory 405 of the electronic device 400 may include any suitable volatile memory, such as a suitable dynamic random access memory (DRAM).

[0099] NVM 406 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, NVM 406 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of a HDD (Hard Disk Drive), a CD (CompactDisc) drive, and a DVD (Digital Versatile Disc) drive.

[0100] NVM 406 may include a portion of storage resources installed on a device of electronic device 400, or it may be accessible by the device but not necessarily a portion of the device. For example, NVM 406 may be accessed over a network via network interface 410.

[0101] In particular, system memory 405 and NVM 406 may include a temporary copy and a permanent copy of instruction 420, respectively. Instruction 420 may include instructions that, when executed by at least one of processors 402, cause electronic device 400 to implement the following: Figure 2In some embodiments, instructions 420, hardware, firmware, and / or software components thereof may additionally / alternatively be placed in system control logic 408, network interface 410, and / or processor 402.

[0102] The network interface 410 may include a transceiver for providing a radio interface for the electronic device 400, thereby communicating with any other suitable device (e.g., a front-end module, an antenna, etc.) through one or more networks. In some embodiments, the network interface 410 may be integrated with other components of the electronic device 400. For example, the network interface 410 may be integrated with at least one of the processor 402, the system memory 405, the NVM 406, and a firmware device (not shown) having instructions. When at least one of the processors 402 executes the instructions, the electronic device 400 implements Figure 2 One or more of the various embodiments shown.

[0103] The network interface 410 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, the network interface 410 may be a network adapter, a wireless network adapter, a telephone modem, and / or a wireless modem.

[0104] In one embodiment, at least one of the processors 402 may be packaged together with one or more controllers for the system control logic 408 to form a system in package (SiP). In one embodiment, at least one of the processors 402 may be integrated on the same die with one or more controllers for the system control logic 408 to form a system on chip (SoC).

[0105] The electronic device 400 may further include an input / output (I / O) device 412 connected to the system control logic 408. The I / O device 412 may include a user interface to enable a user to interact with the electronic device 400; the design of the peripheral component interface enables the peripheral components to interact with the electronic device 400. In some embodiments, the electronic device 400 further includes a sensor for determining at least one of an environmental condition and location information related to the electronic device 400.

[0106] In some embodiments, I / O device 412 may include, but is not limited to, a display (e.g., an LCD display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., an LED flash), and a keyboard.

[0107] In some embodiments, the peripheral component interface may include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.

[0108] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 400. In other embodiments of the present application, the electronic device 400 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0109] Program code can be applied to input instructions to perform the functions described in the present invention and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a system for processing instructions including processor 402 includes any system having a processor such as a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0110] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in the present invention is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0111] According to one embodiment of the present invention, a computer-readable storage medium is further proposed, wherein at least one computer instruction is stored in the computer-readable storage medium, and the at least one instruction is loaded and executed by a processor to implement the aforementioned data processing method based on user-generated content.

[0112] According to one embodiment of the present invention, a computer program product is further provided. The computer program product includes computer instructions. When the computer instructions are executed, the aforementioned data processing method based on user-generated content is implemented.

[0113] Illustrative embodiments of the present invention include, but are not limited to, a method, apparatus, medium, and program product for data processing based on user-generated content.

[0114] The various aspects of the illustrative embodiments will be described using terms commonly used by those skilled in the art to convey the essence of their work to other persons skilled in the art. However, it is apparent to those skilled in the art that some alternative embodiments may be implemented using some of the features described. For purposes of explanation, specific numbers and configurations are set forth so that the illustrative embodiments are more thoroughly understood. However, it is apparent to those skilled in the art that alternative embodiments may be implemented without the specific details. In some other cases, some well-known features are omitted or simplified herein to avoid obscuring the illustrative embodiments of the present invention.

[0115] In addition, various operations will be described as multiple separate operations in a manner that is most helpful for understanding the illustrative embodiments; however, the order of description should not be interpreted as implying that these operations must rely on the order of description, and many of the operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can also be rearranged. When the described operations are completed, the process can be terminated, but can also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0116] References in the specification to "an example", "an example", "an embodiment", "an implementation", etc. indicate that the described embodiment may include a particular feature, structure, or property, but each embodiment may or may not necessarily include the particular feature, structure, or property. Moreover, these phrases are not necessarily directed to the same embodiment. In addition, when particular features are described in conjunction with a specific embodiment, the knowledge of those skilled in the art can affect the combination of these features with other embodiments, whether or not these embodiments are explicitly described.

[0117] Unless the context dictates otherwise, the terms "comprising," "having," and "including" are synonymous. The phrase "A and / or B" means "(A), (B), or (A and B)."

[0118] As used herein, the term "module" may refer to, be part of, or include: memory (shared, dedicated, or group) for running one or more software or firmware programs, application-specific integrated circuits (ASICs), electronic circuits and / or processors (shared, dedicated, or group), combinational logic circuits, and / or other suitable components that provide the described functionality.

[0119] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order is not necessary. Instead, in some embodiments, these features may be described in a manner and / or order different from that shown in the illustrative drawings. In addition, the structural or method features included in a specific figure do not mean that all embodiments need to include such features. In some embodiments, these features may not be included or may be combined with other features.

[0120] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0121] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the intention that the claimed invention requires more features than those expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in less than all of the features of the individual embodiments previously disclosed. Therefore, the claims that follow the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present invention.

[0122] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0123] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

Claims

1. A data processing method based on user-generated content, used in an electronic device, characterized in that: include: An identification step of identifying named entities in the user-generated content and obtaining category words of the named entities; An extraction step of determining a context corresponding to the category word in the user-generated content based on the category word; A recall step, based on the category words, performing a first recall and / or a second recall on the predetermined product category to select candidate items; The sorting step is to calculate the confidence of the candidate items based on the context, and determine the target product category associated with the user-generated content from the candidate items based on the confidence.

2. The data processing method based on user-generated content according to claim 1, characterized in that: The context corresponding to the category word in the extraction step includes the sentence, title, topic and any combination thereof in which the category word is located.

3. The data processing method based on user-generated content according to claim 1, characterized in that: The first recall in the recall step is at least one level of recall based on a predetermined rule.

4. The data processing method based on user-generated content according to claim 3, characterized in that: The at least one level recall includes: First-level recall, the predetermined rule of the first-level recall is that the category word is completely consistent with the leaf node of the commodity category; Secondary recall, the predetermined rule of the secondary recall is that the segmentation result of the category word is completely included in the segmentation result of the leaf node of the commodity category; And, a third-level recall, the predetermined rule of the third-level recall is that more than 50% of the word segmentation results of the category words are included in the word segmentation results of the leaf nodes of the commodity category.

5. The data processing method based on user-generated content according to claim 1, characterized in that: The second recall in the recall step is a token-based vector recall, wherein a token represents the smallest semantic unit of the category word.

6. The data processing method based on user-generated content according to claim 5, characterized in that: Calculating the confidence of the candidate item based on the context includes calculating the confidence of the candidate item based on a twin-tower model, the twin-tower model includes a user-generated content side and a product category side, the input of the user-generated content side includes the context, and the input of the product category side is text in the product database.

7. The data processing method based on user-generated content according to claim 6, characterized in that: The input on the user-generated content side also includes pictures in the user-generated content, and the input on the product category side also includes pictures in the product database.

8. The data processing method based on user-generated content according to claim 6, characterized in that: Calculating the confidence of the candidate item based on the context further includes calculating the confidence of the candidate item based on the context and discrete features of the candidate item.

9. An electronic device, characterized in that: The device comprises a memory storing computer executable instructions and a processor, wherein the processor is configured to execute the instructions to implement the data processing method based on user generated content according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer instruction, and the at least one instruction is loaded and executed by the processor to implement the data processing method based on user-generated content as described in any one of claims 1-8.

11. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed, the data processing method based on user-generated content according to any one of claims 1 to 8 is implemented.

Images