Data standardization method and system and related products

Abstract

The embodiment of the invention discloses a data standardization method and system and a related product, and the method comprises the steps: if the description of a to-be-processed name for at least one naming element does not accord with a corresponding preset description format, determining an error description prompt word for the naming element according to the preset description format; and on the basis of the error description prompt words, description about the naming elements in the to-be-processed name is adjusted, and a new description name conforming to a preset description format is obtained. According to the invention, the system can concentrate on and automatically analyze the error description condition of the name to be processed in the aspect of the naming element aiming at the error description prompt words of the naming element in a thinking-considered manner, so that the time delay and the labor cost caused by a manual intervention process are reduced; and meanwhile, under the condition that the names of the massive articles need to be standardized into new description names, the standardization efficiency and accuracy of the names of the articles are improved, and the related data values of the articles can be conveniently, rapidly and uniformly excavated through the new description names subsequently.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular to data standardization methods, systems and related products. Background Art

[0002] In the process of continuous development of information business, various industries have accumulated a lot of data. However, many of the name data are entered by users themselves, and there is a problem of naming the same item according to their own expression habits. This results in the lack of unified standards for the accumulated data such as item names, and even the easy loss of important information, making it impossible to effectively analyze and mine the value of the data.

[0003] Although the existing data analysis system can perform standardization checks on the entered names, manual intervention is still required to adjust non-standard names to standard names after the check. The entire process is costly and the effect is not ideal. Therefore, the existing data standardization processing is inefficient and has poor accuracy. Summary of the invention

[0004] The embodiments of the present application provide a data standardization method, system and related products for improving the standardization efficiency of non-standard names.

[0005] A first aspect of an embodiment of the present application provides a data standardization method, including:

[0006] Selecting an input name from at least one input name of the same item as a name to be processed;

[0007] If the description of at least one naming element of the name to be processed does not conform to the corresponding preset description format, determine an error description prompt for the naming element according to the preset description format;

[0008] Based on the error description prompt, the description of the naming element in the name to be processed is adjusted to obtain a new description name that conforms to the preset description format.

[0009] Optionally, if there are multiple input names, selecting an input name as a name to be processed from at least one input name of the same item includes:

[0010] Scoring each of the input names in at least one dimension;

[0011] The scoring results of each input name are comprehensively compared, and an input name with the largest comprehensive score is selected as the name to be processed.

[0012] Optionally, the process of generating the error description prompt for the named element includes:

[0013] Inputting at least one input name having the same error cause as the name to be processed into the target model, and obtaining the error description prompt obtained by the target model for the error cause;

[0014] Among them, the error cause refers to that the description of the naming element in the input name does not conform to the preset description format; the error description prompt is at least used to prompt that the attention weight of the information extraction about the naming element in the name to be processed is increased.

[0015] Optionally, before inputting at least one input name having the same error cause as the name to be processed into the target model, the method further comprises:

[0016] A model with a parameter scale greater than or equal to a preset scale is selected as the target model, and the target model is deployed as a cloud application programming interface API, so that the error description prompt is generated by calling the cloud API.

[0017] Optionally, before obtaining the new description name that complies with the preset description format, the method further includes:

[0018] The category of the item is determined based on the name to be processed, so that the selected error description prompt is consistent with the category of the item.

[0019] Optionally, before obtaining the new description name that complies with the preset description format, the method further includes:

[0020] Redundant words in at least one of the input names are removed, and the description of the naming elements in the input name is retained, so as to obtain a simplified name for selection as the name to be processed.

[0021] Optionally, after obtaining the new description name that conforms to the preset description format, the method further includes:

[0022] If the description of at least one of the named elements by the new description name still does not conform to the corresponding preset description format, the new description name is discarded and / or another new description name is obtained.

[0023] When the method described in the first aspect of the present application is implemented, the content described in the second aspect of the present application can be adopted to achieve it.

[0024] A second aspect of an embodiment of the present application provides a data standardization system, including: an acquisition unit and a processing unit;

[0025] The acquisition unit is used to select an input name as a name to be processed from at least one input name of the same item;

[0026] The processing unit is used to determine an error description prompt for the naming element according to the preset description format if the description of at least one naming element of the name to be processed does not conform to the corresponding preset description format;

[0027] The processing unit is further configured to adjust the description of the naming element in the name to be processed based on the error description prompt to obtain a new description name that complies with the preset description format.

[0028] A third aspect of an embodiment of the present application provides an electronic device, including: a processor and a memory;

[0029] The processor is configured to communicate with the memory and execute instructions in the memory to implement the method described in the first aspect of the embodiment of the present application or any specific implementation of the first aspect.

[0030] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, they implement the method described in the first aspect of the embodiment of the present application or any specific implementation of the first aspect.

[0031] A fifth aspect of an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a processor, they implement the method described in the first aspect of the embodiment of the present application or any specific implementation of the first aspect.

[0032] It can be seen from the above technical solutions that the embodiments of the present application have at least the following advantages:

[0033] As the name suggests, the error description prompt for the naming element in the embodiment of the present application allows the system to focus on automatically analyzing the error description of the name to be processed in this naming element, reducing the delay and manpower cost caused by the manual intervention process; at the same time, it helps to improve the standardization efficiency and accuracy of the names of each item when the names of a large number of items need to be standardized into new description names, making it convenient to quickly and uniformly mine the relevant data value of the items through the new description names in the future. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0035] It should be noted that, although the steps in the process diagrams (if any) involved in the embodiments are drawn in sequence as indicated by the arrows, unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0036] Figure 1 A schematic diagram of the system architecture of the data standardization method according to an embodiment of the present application;

[0037] Figure 2 , Figure 3 This is a flow chart of the data standardization method according to an embodiment of the present application;

[0038] Figure 4 A schematic diagram of the structure of the data standardization system of the embodiment of the present application;

[0039] Figure 5 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0041] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims and drawings of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0042] In the following description, similar expressions such as "a specific implementation" or "a specific example" are involved, which describe a subset of all possible embodiments, but it can be understood that "a specific implementation" or "a specific example" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict. In the following description, the term multiple refers to at least two. In some specific examples, the numerical value mentioned in this application reaches the threshold value (if any), which may include the case where the former is greater than the latter of the threshold value; if similar expressions such as "any" or "at least one" are mentioned, it can specifically refer to any one of the listed examples or any combination of these examples.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0044] The method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 101 through the network, and the data storage system 100 can store the data that the server 101 needs to process or needs. The data storage system 100 can be integrated on the server 101, or it can be placed on the cloud or other network servers. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 101 can be implemented as an independent server or a server cluster consisting of multiple servers.

[0045] Terminal 102 can obtain the user-defined name (i.e., input name) for the same item, and send each input name to server 101. For the same item, server 101 can select an input name from at least one input name of the same item as a name to be processed, i.e., a name to be standardized; if the description of at least one naming element in the name to be processed does not conform to the corresponding preset description format, server 101 can adjust the description of the naming element in the name to be processed based on the error description prompt to obtain a new description name that conforms to the preset description format, and the description of the naming element in the new description name conforms to the preset description format more than the name to be processed. Automatically completing the standardized transformation of non-standard names in this way is convenient for subsequent unified and quick mining of the accumulated data value, and optimizing data management and application effects.

[0046] The above-mentioned naming elements may include at least one of the brand name, specification, and quantitative unit of the item. The above-mentioned non-standard name may refer to a name that incorrectly describes the item in at least one naming element. For example, a name with an empty description of the specification or a name that does not conform to the specification description format of "number + unit" can be called a non-standard name.

[0047] It should be noted that the method provided in the embodiment of the present application can be implemented jointly by the terminal device and the server as described above, or it can be implemented entirely on the server side, or it can be implemented entirely on the terminal device side. The specific method can be determined according to the actual application scenario and is not limited here.

[0048] The method of the present application will be further described in detail below, and some specific possible implementation examples will be provided. In actual applications, the implementation contents between these examples can be combined or implemented separately as needed according to the corresponding functional principles and application logic. If implemented in combination, the execution order between the combined examples can be determined according to their respective processing logics, which may be determined by the actual scenario.

[0049] See also Figure 2 In a first aspect, the present application provides a specific embodiment of a data standardization method, which includes the following steps:

[0050] Step S21, selecting an input name from at least one input name of the same item as a name to be processed.

[0051] The above-mentioned items may refer to physical objects such as commodities or documents, and of course may also refer to virtual objects such as electronic images or virtual devices. For the convenience of explanation and understanding, the various embodiments of the present application are mainly explained by taking commodities as an example. The name standardization process of other items can be implemented in the same way as the standardization process of commodity names. The above-mentioned input name may include non-standard names and / or standard names of items. Taking commodities as an example, standardized commodity data (or standard names) generally require at least naming elements such as brands or specifications in addition to the proprietary names of commodities. However, in actual situations, due to various reasons, most of the commodity data (or non-standard names) that are finally entered into the system are non-standardized commodity data (or non-standard names): it may only contain the proprietary names of commodities, or simplified commodity names that only merchants can understand, or it may contain specifications without brands, or brands without specifications, and other various non-uniform data.

[0052] Step S22: if the description of at least one naming element of the name to be processed does not conform to the corresponding preset description format, determine an error description prompt for the naming element according to the preset description format;

[0053] The preset description format corresponding to any naming element may refer to the naming style required for the naming element in factory configuration and / or promotional slogans, etc. For example, the preset description format for the naming element "specification" may be that the description of "specification" is not empty and the description format is "number + unit".

[0054] Still taking the naming element "specification" as an example, if the value (or description) of "specification" extracted from the name to be processed does not conform to the corresponding preset description format, an error prompt involving this preset description format can be found as an error description prompt for "specification"; the "involving" here can mean that this preset description format is recorded in the alternative prompt, or there is a description example that conforms to this preset description format, etc.

[0055] Step S23: Based on the error description prompt, adjust the description of the naming element in the name to be processed to obtain a new description name that complies with the preset description format.

[0056] Still taking the naming element "specification" as an example, the above error description prompt can be obtained by analyzing the input name that also does not conform to the description format of the element "specification" (i.e., the preset description format). When there are multiple input names analyzed here, they may correspond to different or the same items; the content of this error description prompt may include: the reason why the analyzed input name does not conform to the description format of the element "specification" may be that the description of the "specification" is empty and / or the description format is not "number+unit".

[0057] It should be noted that in the embodiment of the present application, when the constituent elements are multiple elements such as "brand, specification, unit", the name conforming to the description format of each element generally means that the description of each constituent element in the current name conforms to the corresponding rules of each constituent element; for example, it conforms to all element description formats under the category to which the item belongs, such as the "specification" of food and beverage must conform to "number + unit", such as any mode of "number + ml, number + gram, number + kg, etc., the "brand" must be a string within a certain length, and the "unit" must be any description value such as individual, bottle, box, etc.

[0058] Still taking the naming element "specification" as an example, it can be seen that the above error description prompt can be used to provide clear prompts, focusing on the "specification" field for identification and extraction, so that this time more attention is paid to extracting the information of the "specification" field from the name to be processed, and adding, deleting or modifying it, so that the description of the new description name in terms of "specification" is better than the standard of the name to be processed, that is, the new description name can be called a standard name compared with the name to be processed.

[0059] In summary, as the name suggests, the error description prompt for the naming element in the embodiment of the present application allows the system to focus on automatically analyzing the error description of the name to be processed in this naming element, reducing the delay and manpower cost caused by the manual intervention process; at the same time, it helps to improve the standardization efficiency and accuracy of the names of each item when the names of a large number of items need to be standardized into new description names, making it convenient to quickly and uniformly mine the relevant data value of the items through the new description names in the future.

[0060] Based on the above examples, Figure 3 Provide some concrete examples of possible implementations.

[0061] Generally, each commodity is assigned a corresponding barcode (or barcode for short), so commodities with the same barcode can be summarized according to the barcode, so that items with different input names but the same commodity can be distinguished. For example, there is a barcode 123456789000, which is assigned to a specific commodity, such as a bottle of a certain brand of mineral water. No matter whether the bottle is 500 ml or 1 liter, as long as its barcode is 123456789000, by scanning this barcode, the mineral water of this brand and its input name can be found in the system. These input names can be custom names (including standard names) entered by different merchants according to language habits. For example, the barcode 4006381333900 can be used to uniquely identify a product. Suppose this barcode is assigned to a smart phone. No matter how the color, storage capacity or other configuration of the phone changes, as long as its barcode 4006381333900 remains unchanged, scanning this barcode will point to the associated information of the same product, such as summarizing the phone and its input name, that is, separating one-to-many (that is, the same item has multiple names) product names.

[0062] In some examples, a rule base may be prepared or constructed first. The preset description format corresponding to each naming element may be recorded in the rule base in the form of a regular expression, etc. These preset description formats may be classified according to the category of the item (such as food, clothing, etc.). The rule base may specifically include a specification library, a brand library, and a unit library. The brand library may contain most of the common brands currently on the market.

[0063] The above-screened names to be processed contain at least the description content of a preset number of naming elements. In other words, the screened names to be processed can be a name record that contains as much valid information as possible, because the more valid information, the easier it is to generate standard names quickly and accurately in the future, thereby reducing resource loss and waiting time; the above-mentioned valid information can refer to the element values ​​corresponding to the naming elements, such as "50ml" (i.e. specification value) and "XY" (i.e. brand name). Therefore, based on Figure 2In some specific examples, if there are multiple input names for the same item, the specific operation process of step S21 may include (extracting information for matching scoring): scoring each input name in at least one dimension; comprehensively comparing the scoring results of each input name, and selecting the input name with the largest comprehensive score as the name to be processed.

[0064] Exemplarily, from multiple input names of the same item, an input name with the best score result can be selected as the name to be processed to be standardized. Exemplarily, the information of each component element (such as the three elements of brand, specification and unit) of each input name can be extracted for scoring in at least one dimension such as rule matching and similarity calculation, wherein the input name that meets all the rules can be directly recorded as a standard name and put into the library (such as falling into the standard name library), and may not participate in the subsequent standardization operation; and the input name that does not meet all the rules (i.e., non-standard name) may be separated in whole or in part as a name to be processed, to be processed twice (such as executing step S23) into a standard name; if a part is selected, the non-standard name with the largest total score can be selected as the name to be processed. In short, the name to be processed for the same item can be one or more, and the standard name finally retained for the same item can be one or more, wherein the name that describes the three elements in compliance can be called the standard name.

[0065] This rule matching (or pattern matching) may refer to that the description of a certain naming element by the current input name complies with the preset description format of the element. For example, if the description of "specification" by the current input name is not empty and complies with the description format of "number + unit", it can be said that it successfully matches the element of "specification", and x points (such as +5 points) can be added; on the contrary, if it does not comply with the description format of this element (such as the value of the element of "specification" is empty), no points will be added. For another example, if the description of "brand" by the current input name can be successfully matched with an existing brand in the brand library, that is, it belongs to an existing brand, then the current input name can be added with y points (such as +5 points). In this way, the total rule matching score of the current input name under each component element can be calculated, and the total rule matching score can be up to Z points (such as 15 points).

[0066] The similarity score can refer to the similarity score between the current input name and other input names of the same item. According to the experience or theory of "the minority is subject to the majority", the higher the commonality between the two names, the higher the similarity. To a certain extent, it can be said that the more unified the two names involved in the comparison are, the more likely the current input name is to tend to the structure required by the standard name (such as brand + specification + unit), that is, the more likely it is to meet all the element description formats and serve as a standard name. Therefore, the higher the similarity between the two names (such as cosine similarity or Pearson correlation coefficient), the higher the similarity score of the current input name.

[0067] Finally, the above-calculated scores, i.e., the matching scores corresponding to each factor + the cosine similarity scores, can be combined to select the input name with the largest total score as the name to be processed, and the name to be processed selected at this time can be regarded as the optimal name containing more valid information. It should be noted that the cosine similarity score and the matching scores corresponding to each factor can be the same (such as x can be equal to y) or different, without specific limitation.

[0068] Alternatively, as another possible implementation, a random input name can be selected as the name to be processed. In addition, the same item can have one or more names to be processed, which can be determined by actual conditions and are not limited here.

[0069] based on Figure 2 In some specific examples, in order to avoid wasting computing resources and feedback delays, before step S23, the method of the embodiment of the present application may also include (simplifying the input name): removing redundant words in at least one input name, and retaining the description of the naming elements in the input name, to obtain a simplified name for selection as the name to be processed.

[0070] Exemplarily, redundant words such as product promotion modifiers, additional words unrelated to the product's proprietary name or the three elements, etc. can be removed from the input name. For example, for the input name: "Paris XYZ Qihuan Essential Oil Shampoo (Qihuan Chuishun) 440ml / (subject to the actual product in the store) bottle", the optimized name obtained after simplification can be: "Paris XYZ Qihuan Essential Oil Shampoo 440ml / bottle", which removes unnecessary descriptive language and retains the original field information of the three elements of the input name, such as brand, specification and unit. It should be noted that the simplified name can be the input name before screening, or the screened name to be processed, that is, the simplification operation can be implemented before or after step S21, which can be set according to the actual situation and is not limited here.

[0071] based on Figure 2 In some specific examples, the generation process of the error description prompt in step S22 may include: inputting at least one input name with the same error cause as the name to be processed into the target model, and obtaining an error description prompt obtained by the target model for the error cause; wherein the error cause refers to the description of the naming element in the input name not conforming to the preset description format; the error description prompt is at least used to prompt that the attention weight of the information extraction about the naming element in the name to be processed is increased, which can be simply understood as making the focus on or focusing on extracting information about this naming element.

[0072] Exemplarily, a new description name of an item can be generated with the help of a large language model (LLM). Compared with a small model, this LLM (or large model) can refer to an advantageous model that increases the model size (such as parameter scale), training data, and computing resources. This LLM can be a language model of the Qwen series or the GLM series; this Qwen is often pre-trained based on a massive data set of trillions of entries, and uses supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) technology to make the model conform to human preferences. In actual situations, there are name standardization LLMs for standardization tasks, classification LLMs for classification, three-element extraction LLMs for extracting element information, and prompt optimization LLMs for extracting prompts. These LLMs can be the same or different, such as the parameter scales can be the same or different.

[0073] From the input names that do not conform to the description format of each element (which can be the same or different items), a part of the input names can be extracted as error analysis samples. Taking the element description format that does not conform to the "specification" as an example, the input names that do not conform to this rule can be extracted and provided to the target model (such as LLM), and the target model can obtain the error description prompts refined for the error cause; because in some cases, there are the same error causes as references, which is convenient for LLM to solve similar problems quickly and accurately.

[0074] For example, for the error analysis sample with the input name "XY Wild Blueberry Brandy", the error output in terms of the three elements of "brand, specification, unit" is: [XY][Wild Blueberry][Brandy], that is, [XY][Wild Blueberry][Brandy] is identified as the corresponding description values ​​of the three elements of "brand, specification, unit", but such an identification result itself is an error. Through LLM's own understanding ability, LLM can correctly output this error situation: [XY][][], where "[][]" means that the description (or value) of "specification, unit" is empty. Further, LLM can abstract the error description prompt (which can be called the error analysis result) for this error situation: "Wild Blueberry" is mistakenly identified as a specification, but in fact it is a characteristic description of the product; "Brandy" is a product type rather than a unit.

[0075] In actual situations, the above error description prompt can provide a reference for LLM to understand other input names, helping to enhance LLM's understanding ability; on the other hand, the error description prompt in the above example implicitly prompts "Please try to extract information from the name to be processed again, this time focusing on extracting the "specification" and "unit" field information", so the error description prompt can be called a new extraction prompt. It can be seen that the error description prompt can instruct LLM to focus on extracting information from elements that do not conform to the element description format, so that the names to be processed that do not conform to the element description format can be processed into standard names, that is, the error description prompt is equivalent to providing the names to be processed with an opportunity to try to be standardized. It should be supplemented that the primary processing relative to the secondary processing here may refer to the processing process when the user inputs a name or the system automatically filters out the names to be processed.

[0076] Of course, there can be multiple error description prompts for the same element. Specifically, multiple rounds of dialogue can be conducted by switching different error description prompts, allowing LLM to better focus on field extraction for this element. In some examples, the above error description prompts can be abstracted into corresponding regular expressions and stored in the rule library as verification rules for verifying whether the input name conforms to the element description format.

[0077] In some specific examples, for low cost considerations, before the above-mentioned "inputting at least one input name with the same error cause as the name to be processed into the target model", the method of the embodiment of the present application may also include (deploying the model service in the form of a cloud API): selecting a model with a parameter scale greater than or equal to a preset scale as the target model, and deploying the target model as a cloud application programming interface API, so that an error description prompt is generated by calling the cloud API.

[0078] For example, an LLM with a size greater than or equal to a preset size (such as 7b) can be selected as the target model. 7b (b stands for billion) refers to the number of trainable parameters in the model, which is 7 billion. The larger the parameter size of the LLM, the stronger its generalization and understanding capabilities are, and the better it can output ideal results. For another example, if the system hardware has sufficient support for LLM, you can consider using an LLM with a larger parameter, such as a 72b LLM as the target model.

[0079] On the other hand, considering the high cost of local deployment and training of LLM, such as the need to manually annotate a large number of training samples to train LLM, in actual situations, you can choose to call the cloud API to allow LLM to generate error description prompts for the above error situations. In other words, LLM can be deployed as an online API service, providing services to the outside world through the network in the form of an API architecture. Specifically, this LLM can be accessed through the HTTPS protocol; this deployment method allows LLM to be called externally as a service without exposing the entire model system to the caller, which can reduce the cost of local deployment or training of the model for the caller. Of course, it is also possible to evaluate the existing large models on the market and select large models that can be deployed and maintained locally and that can implement this method based on time and cost.

[0080] It can be seen that the embodiment of the present application can utilize the recognition capability of LLM to generate a new description name that is more in line with the standards based on the existing product name. The composition structure of this new description name includes the following three elements: brand + specification + unit. The accuracy (or degree of standardization) reflected by the new description name is higher.

[0081] based on Figure 2 In some specific examples, before step S23, the method of the embodiment of the present application may further include (classifying the names to be processed): determining the category of the item based on the name to be processed, so that the selected error description prompt is consistent with the category of the item.

[0082] Exemplarily, the name to be processed can be input into the big model, so that the big model can specify the item category to which the name to be processed belongs in a given item classification list (including food, clothing, electronic products, etc.), that is, the current item is classified into a specific classification; this classification can include primary, secondary, and tertiary classifications (the classification granularity becomes finer level by level), for example, the secondary category under the primary category "food" can include "beverages", and the tertiary category under it can include "carbonated beverages", etc. Distinguishing the item category here helps to accurately select the prompt words, element description formats, etc. under the category according to the item category in the subsequent relevant operations, so that the implementation of each operation is more in line with the actual item category, and the accuracy of data standardization is guaranteed to be improved. In layman's terms, classifying categories helps to avoid the element description format of food (such as 50mg, 50ml*2 and other reference specifications) being mistakenly cited as the element description format of electronic products (actually should be 256GB, 1TB and other reference specifications), resulting in incorrect operation results.

[0083] Specifically, because the standards of each industry are different, for example, most food and beverage products have specifications such as capacity, size, bottle, box, etc., while clothing products have specifications such as size, color, material, etc., so a corresponding element description format can be formulated for each industry category. Similarly, an effective summary can be made based on the industry characteristics of each industry to write prompts (i.e., prompts) that adapt to the industry standards, that is, prompts provided to the big model to accurately understand and apply the industry standards.

[0084] As described above, a standardized prompt for standardization can be written and optimized to perform a name standardization task; similarly, in some examples, a classification prompt for classification can be written and optimized to perform a product classification task, which will not be described in detail.

[0085] based on Figure 2 In some specific examples, after step S23, the method of the embodiment of the present application may further include (accuracy evaluation and reprocessing): if the description of at least one naming element by the new description name still does not conform to the corresponding preset description format, discard the new description name and / or obtain other new description names.

[0086] After each name to be processed that does not conform to the description format of each element has undergone a round of step S23 (i.e., after prompt optimization processing) and other operations, a part of it can finally become in line with the description format of each element, that is, the new description name obtained after one processing can be stored in the library as a standard name. As for the other part of the names to be processed that still do not conform to the description format of each element after a round of processing in step S23, these new description names obtained after one processing can be discarded and / or the operations such as step S23 can be repeated. Here, repeating the operations such as step S23 is equivalent to trying again to process to obtain a new description name that conforms to the description format of each element, ensuring that each name to be processed can be standardized to a certain extent, thereby providing convenience for item management and registration; the number of repetitions, etc. can be determined according to the situation and is not limited here.

[0087] In actual situations, the process of repeatedly executing step S23 may include fine-tuning the large model to enhance the model's adaptability to the names to be processed that do not conform to the element description format. The process of fine-tuning the large model may be to increase the parameter scale of the model and / or retrain the model. For example, a 7b LLM may be used when executing step S23 last time, and when hardware resources are sufficient, a larger scale such as 72b LLM may be used when executing step S23 again.

[0088] Exemplarily, the quality of the output data (i.e., the name) can be evaluated based on existing rules, and non-compliant data can be separated for optimization. For example, the accuracy of the new description name obtained after processing in step S23 can be calculated to evaluate the degree of standardization of the new description name: the new description name can add x points for each element description format it meets, and the accuracy of the new description name = the total score obtained by the new description name under each element description format / the full score when it meets all element description formats. If the accuracy of the new description name is lower than a certain threshold, it can indicate that the degree of standardization of the new description name is still not high enough, and you can try to return to step S23 to obtain other new description names again.

[0089] In summary, the embodiments of the present application can combine the prompts and the LLM's own understanding ability to perform data (i.e., item names) standardization processing while minimizing costs, greatly reducing time delays and labor costs, and at the same time improving the accuracy of the data after standardization. Figure 3 As shown, the method of the embodiment of the present application, in the preprocessing stage: a scoring system can be designed for multi-name products to optimize the input quality of the names to be processed; in the processing stage: combined with the use of prompt engineering, the accuracy of data processing can be improved, wherein LLM can be used to automatically optimize prompts and perform secondary processing on names that do not meet the rules, in order to try again to improve the accuracy of name standardization.

[0090] Compared to Figure 2 The examples shown illustrate that the above-mentioned additional or detailed example contents or possible implementation methods do not necessarily have to be executed during the specific implementation. If more than two examples or possible implementation methods are added, these examples or possible implementation methods can be implemented in combination or separately. If implemented in combination, the execution order between the combined examples can be determined according to their respective processing logic, which can be determined according to the actual scenario.

[0091] See also Figure 4 , the second aspect of the present application provides a specific embodiment of a data standardization system, the system comprising: an acquisition unit 401 and a processing unit 402;

[0092] The acquisition unit 401 is used to select an input name from at least one input name of the same item as a name to be processed;

[0093] The processing unit 402 is used to determine an error description prompt for the naming element according to the preset description format if the description of at least one naming element of the name to be processed does not conform to the corresponding preset description format;

[0094] The processing unit 402 is further configured to adjust the description of the naming element in the name to be processed based on the error description prompt to obtain a new description name that complies with a preset description format.

[0095] In some examples, if there are multiple input names, the obtaining unit 401 is specifically used to:

[0096] Score each input name on at least one dimension;

[0097] The scoring results of each input name are comprehensively compared, and the input name with the largest comprehensive score is selected as the name to be processed.

[0098] In some examples, the processing unit 402 is specifically configured to:

[0099] Inputting at least one input name having the same error cause as the name to be processed into the target model, and obtaining an error description prompt for the error cause obtained by the target model;

[0100] The error reason refers to that the description of the naming elements in the input name does not conform to the preset description format; the error description prompt is at least used to prompt and increase the attention weight on the information extraction about the naming elements in the name to be processed.

[0101] In some examples, the processing unit 402 is further configured to:

[0102] A model with a parameter scale greater than or equal to a preset scale is selected as a target model, and the target model is deployed as a cloud application programming interface API, so that an error description prompt is generated by calling the cloud API.

[0103] In some examples, the processing unit 402 is further configured to:

[0104] The category of the item is determined based on the name to be processed, so that the selected error description prompt is consistent with the category of the item.

[0105] In some examples, the processing unit 402 is further configured to:

[0106] Redundant words in at least one input name are removed, and descriptions of naming elements in the input name are retained, so as to obtain a simplified name for selection as a name to be processed.

[0107] In some examples, the processing unit 402 is further configured to:

[0108] If the description of at least one naming element by the new description name still does not conform to the corresponding preset description format, the new description name is discarded and / or another new description name is obtained.

[0109] In the embodiment of the present application, the operations performed by each unit of the data standardization system are similar to the operations described in the first aspect or any specific method embodiment of the first aspect, and will not be described in detail here. Of course, the specific implementation process of each operation in the first aspect of the present application can also refer to the relevant description of the second aspect.

[0110] See also Figure 5 The electronic device 500 of the embodiment of the present application may include one or more processors (such as central processing units (CPU)) 501 and a memory 505, in which one or more applications or data are stored.

[0111] The memory 505 may be a volatile storage or a persistent storage. The program stored in the memory 505 may include one or more modules, each of which may include a series of instruction operations in the electronic device. Furthermore, the processor 501 may be configured to communicate with the memory 505 and execute a series of instruction operations in the memory 505 on the electronic device 500.

[0112] The electronic device 500 may also include one or more power supplies 502, one or more wired or wireless network interfaces 503, one or more input and output interfaces 504, and / or one or more operating systems, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0113] The processor 501 can execute the operations performed by the aforementioned first aspect or any specific method embodiment of the first aspect, and the details will not be repeated here.

[0114] The present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enable the computer to execute the method described in the first aspect or any specific implementation of the first aspect.

[0115] The present application provides a computer program product comprising instructions or a computer program. When the computer program product is run on a computer, the computer is enabled to execute the method described in the first aspect or any specific implementation of the first aspect.

[0116] It is understood that in various embodiments of the present application, the sequence number of each step does not mean the order of execution, and the execution order of each step should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The operation content added or refined in each example scheme of the above method, system or device (if any) does not necessarily have to be executed in the specific implementation. If more than two operations are added, these operations can be implemented in combination or separately, depending on the actual scenario.

[0117] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system (if any) and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0118] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system or device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0119] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0120] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0121] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product (or computer program product) is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a business server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A data standardization method, characterized in that: include: Selecting an input name from at least one input name of the same item as a name to be processed; If the description of at least one naming element of the name to be processed does not conform to the corresponding preset description format, determine an error description prompt for the naming element according to the preset description format; Based on the error description prompt, the description of the naming element in the name to be processed is adjusted to obtain a new description name that conforms to the preset description format.

2. The data standardization method according to claim 1, characterized in that: If there are multiple input names, selecting an input name from at least one input name of the same item as a name to be processed includes: Scoring each of the input names in at least one dimension; The scoring results of each input name are comprehensively compared, and an input name with the largest comprehensive score is selected as the name to be processed.

3. The data standardization method according to claim 1, characterized in that: The determining, according to the preset description format, an error description prompt for the named element includes: Inputting at least one input name having the same error cause as the name to be processed into the target model, and obtaining the error description prompt obtained by the target model for the error cause; Among them, the error cause refers to that the description of the naming element in the input name does not conform to the preset description format; the error description prompt is at least used to prompt that the attention weight of the information extraction about the naming element in the name to be processed is increased.

4. The data standardization method according to claim 3, characterized in that: Before inputting at least one input name having the same error cause as the name to be processed into the target model, the method further comprises: A model with a parameter scale greater than or equal to a preset scale is selected as the target model, and the target model is deployed as a cloud application programming interface API, so that the error description prompt is generated by calling the cloud API.

5. The data standardization method according to claim 1, characterized in that: Before obtaining the new description name that conforms to the preset description format, the method further includes: The category of the item is determined based on the name to be processed, so that the selected error description prompt is consistent with the category of the item.

6. The data standardization method according to claim 1, characterized in that: Before obtaining the new description name that conforms to the preset description format, the method further includes: Redundant words in at least one of the input names are removed, and the description of the naming elements in the input name is retained, so as to obtain a simplified name for selection as the name to be processed.

7. The data standardization method according to any one of claims 1 to 6, characterized in that: After obtaining the new description name that conforms to the preset description format, the method further includes: If the description of at least one of the naming elements by the new description name still does not conform to the corresponding preset description format, the new description name is discarded and / or another new description name is obtained.

8. A data standardization system, characterized in that: include: Acquisition unit and processing unit; The acquisition unit is used to select an input name as a name to be processed from at least one input name of the same item; The processing unit is used to determine an error description prompt for the naming element according to the preset description format if the description of at least one naming element of the name to be processed does not conform to the corresponding preset description format; The processing unit is further configured to adjust the description of the naming element in the name to be processed based on the error description prompt to obtain a new description name that complies with the preset description format.

9. An electronic device, characterized in that: include: Processor and memory; The processor is configured to communicate with the memory and execute instructions in the memory to implement the method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that The computer program product comprises computer instructions, which implement the method according to any one of claims 1 to 7 when executed by a processor.

Images