Data processing method and apparatus, computer device, storage medium, and product

Abstract

A data processing method, applied to the field of artificial intelligence. The method comprises: acquiring data and a corresponding label, the label corresponding to an expert rule; and under the condition that the number of data sets corresponding to the label is smaller than a first threshold value, inputting into an algorithm model prompt information composed of the data sets, the label, and the expert rule to generate new data corresponding to the label, wherein the prompt information comprises the label, the data, and the expert rule; and there is a mapping relationship among the label, the data, and the expert rule. According to the present application, the prompt information prompt comprising the expert rule is used as the input of the algorithm model, the algorithm model can be guided to generate new data so as to expand small sample data, and the expanded small sample data set is fused with a preprocessed large sample data set, realizing the balance of data sets of different labels. Meanwhile, under the condition that the diversity and balance of the data are ensured, the performance of the model is improved by combining the dynamic optimization of the prompt information prompt.

Description

Data processing method and device, computer device, storage medium and product

[0001] The present application claims priority from the Chinese patent application No. 202410458799.2 filed on April 16, 2024, and entitled "A data processing method, device, computer device, storage medium and product", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a data processing method, device, computer product and computer storage medium. BACKGROUND

[0003] Machine learning (ML) is a method for realizing artificial intelligence (AI), and the goal of the machine learning method is to design and analyze some algorithms (i.e., models) that allow computers to automatically "learn". The algorithm involved is called a machine learning model. The machine learning model is a kind of algorithm that automatically analyzes rules from data and uses the rules to predict unknown data. Machine learning algorithms are widely used in many fields, such as label classification, image recognition, natural language processing, etc. However, too small sample data may affect the classification / recognition ability of the algorithm model; too large sample data may cause overfitting problems. In the application of machine learning models, the problem of sample imbalance is often encountered, which means that in a limited data set, the number of samples of different categories is very different, resulting in a large number of majority class samples and a small number of minority class samples. This sample imbalance may seriously affect the learning effect of the model, causing the model to tend to predict the majority class and ignore the minority class, affecting the accuracy of the model application.

[0004] Therefore, how to improve the balance of data has become a problem to be solved. SUMMARY

[0005] The present application provides a data processing method, which improves the diversity and balance of generated data, combines dynamic optimization of prompt information, and thus improves the performance of the algorithm model.

[0006] In a first aspect, the present application provides a data processing method. The data processing system first acquires a small sample data set from a data set and a first label from a plurality of labels, wherein the small sample data set is original data to be processed with a data volume less than a first threshold, and the first label is a label associated with the small sample data set, and the first label is used to indicate an attribute of the data; then determines a first data set according to the first label and an expert rule corresponding to the first label, wherein the first data set includes the small sample data set and expansion data obtained based on expansion of the small sample data set, and the expert rule includes an expert rule description of an expert experience knowledge system in a specific field associated with the first label, and the expert rule description includes a text representation of at least one of a term associated with the data, a constraint condition, a data format, and a logical relationship; then acquires a large sample data set from a data set stored in the data processing system, and performs preprocessing on the large sample data set to obtain a second data set, wherein the data volume of the large sample data set is greater than or equal to a second threshold; and finally, acquires a balanced sample data set according to the first data set and the second data set.

[0007] As can be known from the above method description, the present application integrates expert knowledge of an enterprise or organization in a specific field into prompt prompt information, generates expansion data with specific scene characteristics according to the label corresponding to the data and the expert rule, effectively expands the small sample data to obtain the first data set; and further utilizes the preprocessing of the large sample data to obtain the second data set, and then obtains the final balanced sample data based on the first data set and the second data set. Thus, by utilizing the label and the expert rule to expand the small sample data, the model's understanding ability for the small sample data is improved by adding the expert rule in the prompt information, so as to generate accurate and effective small sample data, which not only expands the data volume of the small sample data, increases the diversity of the data, and improves the balance of the sample data.

[0008] In a possible implementation, before acquiring the small sample data set and the first label, a mapping relationship among the label, the expert rule, and the data set can be constructed. Specifically, the original data set, the label library, and the expert rule library can be acquired first, wherein the original data set includes a plurality of data sets with different data attributes, the label library includes labels used to indicate different data attributes, and the expert rule library includes a plurality of expert rules and logical relationships among the plurality of expert rules; then a mapping relationship among the data, the label, and the expert rule is constructed according to the data attribute, and the mapping relationship among the data set, the label, and the expert rule is used to indicate the mapping relationship among the label, the expert rule, and the plurality of data sets. By pre-construction of the mapping relationship among the label, the expert rule, and the data set, the rule library is constructed by utilizing the expert experience in the specific field, and the correspondence among the label, the expert rule, and the data set is determined based on the data attribute, thereby improving the accuracy of expansion of the small sample data.

[0009] In a possible implementation, the first label and the expert rule corresponding to the first label are taken as first prompt information, and the first data set is determined according to the first prompt information; the first prompt information belongs to prompt information, and the prompt information includes any one or more of data, expert rules and labels.

[0010] In a possible implementation, the first data set is determined according to the first label and the expert rule corresponding to the first label, and further includes: generating the expansion data corresponding to the first label and based on the small sample data set according to the first label and the expert rule corresponding to the first label; in a case where the number of the expansion data reaches a third threshold, the expansion data generated and the small sample data set are fused to obtain the first data set.

[0011] In a possible implementation, the pre-processing is performed on the large sample data set to obtain the second data set, and further includes: performing under-sampling on the large sample data set until the number of data in the large sample data set reaches a fourth threshold; taking the data set that has undergone the under-sampling processing and has the number of data reaching the fourth threshold as the second data set.

[0012] In a possible implementation, the balanced sample data set is obtained, including: respectively acquiring a first sampling weight corresponding to the first data set and a second sampling weight corresponding to the second data set; determining the balanced sample data set according to the first data set and the first sampling weight corresponding to the first data set, and the second data set and the second sampling weight corresponding to the second data set.

[0013] In a possible implementation, the label corresponds to the rule; the label includes a first-level label, a second-level label, a third-level label, …, and an Nth-level label, and the rule includes a first-level rule, a second-level rule, a third-level rule, …, and an Nth-level rule.

[0014] The Nth-level label corresponds to the Nth-level rule, and the Nth-level rule includes the first-level rule, the second-level rule, the third-level rule, …, and an (N-1)th-level rule; N is a positive integer.

[0015] In a possible implementation, the data and the labels corresponding thereto are stored in a database; the rule is stored in a rule library, and the rule library can perform one or more operations of querying, editing and updating.

[0016] In a possible implementation, the updating operation includes: updating the rule corresponding to the label according to a review result of the expansion data obtained based on the small sample data set expansion to obtain an updated rule; and updating the prompt information according to the updated rule.

[0017] In one embodiment, the method of reviewing the augmented data based on the small sample data set comprises manual review or review based on similarity of the augmented data to the first label.

[0018] In a second aspect, the present application provides a data processing system, comprising modules for performing the data processing method of the first aspect or any possible implementation of the first aspect.

[0019] In a third aspect, the present application provides a computer device, comprising a processor, a memory, a communication interface, and a bus, the processor, the memory, and the communication interface are connected through the bus and complete communication with each other, the memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions in the memory to perform the operation steps of the method of the first aspect or any possible implementation of the first aspect by using hardware resources of the computer device.

[0020] In a fourth aspect, the present application further provides a computer device cluster, comprising a plurality of computer devices as described in the third aspect, each computer device comprising a processor, a memory, a communication interface, and a bus, the processor, the memory, and the communication interface are connected through the bus and complete communication with each other, the memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions in the memory to perform the operation steps of the method of the first aspect or any possible implementation of the first aspect by using hardware resources of the computer device.

[0021] In a fifth aspect, the present application provides a chip system, comprising a processor and a power supply circuit, the power supply circuit is configured to supply power to the processor, and the processor is configured to perform the operation steps of the method of the first aspect or any possible implementation of the first aspect.

[0022] In a sixth aspect, the present application provides a computer-readable storage medium, comprising a computer program stored in the computer-readable storage medium, and the computer program is configured to perform the operation steps of the method of the first aspect or any possible implementation of the first aspect when executed by a processor.

[0023] In a seventh aspect, the present application provides a computer program product, comprising computer-readable instructions stored in the computer program product, and the computer-readable instructions are configured to perform the operation steps of the method of the first aspect or any possible implementation of the first aspect when executed by a processor.

[0024] On the basis of the implementation manners of the above aspects provided by the application, further combinations can be made to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0025] Fig. 1 is a structural schematic diagram of a data processing system provided by the application;

[0026] Fig. 2 is a flow schematic diagram of a data processing method provided by the application;

[0027] Fig. 3 is a flow schematic diagram of an expert rule dynamic updating method provided by the application;

[0028] Fig. 4 is a structural schematic diagram of a data processing computing device provided by the application. DETAILED DESCRIPTION

[0029] In order to facilitate understanding, first, the terms involved in the application are introduced.

[0030] (1) Data

[0031] Data is used to represent data in the field of artificial intelligence, and the field and form of data are not limited herein. For example, data includes but is not limited to graphics, images, speech, text, video, a mixture of speech and text data, a mixture of two or more different forms of data, and the form of data is not limited herein. The field involved in the data includes but is not limited to the fields of finance, law, smart home, power, medical treatment, intelligent manufacturing, education, and the like, and the fields are not limited herein. The bearing mode of data includes but is not limited to financial data, including transaction data, pricing data, accounting data, tax data, investment data, and the like. It also involves Internet of Things data of traditional devices, including business data of existing systems and sensing data such as force, displacement, liquid level, temperature, humidity, and the like. Here, enumeration is not performed, and the data is not limited.

[0032] (2) Data set

[0033] A data set is used to represent a collection of a group of sample data, and in the application, it is especially used to represent a collection of a group of sample data in the field of artificial intelligence. A group of sample data includes one or more features, and in addition, can contain labels or not. In the field of artificial intelligence, a data set can include a training set, a test set, and a validation set.

[0034] (3) Data processing

[0035] Data processing generally includes data preprocessing, data training, machine learning, deep learning, search, reasoning, decision, and the like.

[0036] Among them, machine learning and deep learning can model, extract, preprocess, train, etc. of symbolic and formalized intelligent information. Reasoning refers to the process of simulating human intelligent reasoning methods in a computer or intelligent system, using formalized information to solve problems with machine thinking and reasoning control strategy. The typical function is search and match. Decision-making refers to the process of decision-making after intelligent information is reasoned. It usually provides classification, sorting, prediction and other functions.

[0037] (4) Prompt

[0038] Prompt refers to a piece of text or an instruction that guides the model to generate output as input. For example, Prompt is a piece of text designed to guide the model to produce meaningful output when performing a specific task. For example, Prompt can be a sentence, a paragraph, or even a complete question.

[0039] Prompt is commonly used in various deep learning tasks, including natural language processing, computer vision, and reinforcement learning. In the field of natural language processing, Prompt is used in text generation tasks such as language model fine-tuning, text summarization, translation, and question and answer systems. Through Prompt, the model can be guided to generate text in a specific field or style. For example, Prompt can be designed to accurately reflect the task requirements and industry characteristics by combining human expert knowledge or domain expert experience, thereby improving the professionalism and practicality of the model. In the field of computer vision, Prompt can also be used in image generation, image classification, and object detection tasks. By providing accurate guidance information to the model, the accuracy and generalization ability of the model can be improved. When training and fine-tuning deep learning models, a well-designed Prompt can significantly improve the performance and output quality of the model.

[0040] (5) Label

[0041] Generally speaking, in multi-classification tasks in machine learning, "label" refers to the correct output or category of an instance, also known as target variable or response variable. Typically, a dataset contains two parts: features and labels. Features are used to describe the properties or characteristics of instances, while labels are used to train and evaluate machine learning models. For example, in image classification problems, features are pixel values or feature descriptors, and labels are categories of images such as cats, dogs, or birds.

[0042] In machine learning, especially in supervised learning, labels refer to the data results output by the model. In this application, labels include data results output by the model, manually annotated keywords, and categories corresponding to samples, which are not listed here. A sample data can correspond to multiple labels.

[0043] (6) Small sample dataset

[0044] A set of data whose data quantity is less than a certain threshold. The dataset can be any of the training set, test set, validation set.

[0045] (7) Large sample dataset

[0046] A set of data whose data quantity is greater than a certain threshold. The dataset can be any of the training set, test set, validation set.

[0047] In order to improve the balance of data, the present application provides a data processing method, which first expands the small sample dataset to obtain a first dataset, then pre-processes (such as under-sampling) the large sample dataset to obtain a second dataset, and finally obtains a balanced sample dataset according to the first dataset and the second dataset. The data processing process specifically includes the following processes: first, a small sample dataset and a first label are obtained, wherein the small sample dataset is the original data to be processed whose data quantity is less than a first threshold, and the first label is a label associated with the small sample data, and the first label is used to indicate an attribute of the data. Second, the first dataset is determined according to the first label and expert rules, the first dataset includes the first small sample dataset and the expansion data obtained based on the expansion of the first small sample dataset, and the expert rules include the expert rule description of the expert experience knowledge system in a specific field associated with the first label, and the expert rule description includes the text representation form of the terms, constraints, data format, and logical relationship associated with the data. A large sample dataset is obtained, and a second dataset is obtained by pre-processing the large sample dataset. The large sample dataset is a dataset stored in the data processing system, and the data quantity in the large sample dataset is greater than or equal to a second threshold. Finally, a balanced sample dataset is obtained according to the first dataset and the second dataset. Thus, the expanded small sample data is obtained by means of the label and the expert rule, the understanding ability of the model for the small sample data is improved by adding the expert rule in the prompt information, so as to generate accurate and effective small sample data, which not only expands the data quantity of the small sample data, increases the diversity of the data, and improves the balance of the sample data. On the other hand, the expert knowledge (expert rule) of the enterprise or organization in a specific field is integrated into the prompt information (prompt), and data representative of a specific scene is generated, thereby improving the accuracy of the index prediction result.

[0048] The data processing method provided by the present application will be described in detail below with reference to the accompanying drawings.

[0049] Fig. 1 is a structural schematic diagram of a system to which the data processing method provided by the present application is applied, as shown in the figure, the system includes a client 100, a data processing system 200, and a storage system 300.

[0050] The client 100 is configured to implement human-computer interaction and can be deployed on a terminal device, which includes a personal computer, a smartphone, a wearable device, a palm-held processing device, a tablet computer, a mobile notebook, an augmented reality (AR) device, a virtual reality (VR) device, an integrated game console, a wearable device, a vehicle-mounted device, a smart conference device, a smart advertising device, a smart home appliance, and the like. The smart home appliance can be a sweeping robot, a mopping robot, or the like, and is not specifically limited herein.

[0051] In specific implementations, the client 100 can be software or an application program running on a terminal device or a computing device controlled by a user, such as a personal computer (PC) client, a web client based on a browser, an application (APP) client running on a mobile terminal, or a console of a cloud platform, and is not specifically limited herein.

[0052] In specific implementations, the client 100 is applied to fields with data analysis and query requirements, such as data analysis, data mining, data exploration, and data query, including but not limited to the biomedical field (such as epidemiological research and drug development), the financial field (risk assessment and market prediction), the social media field (personalized recommendation, advertisement positioning, and sentiment analysis), the sales field (pricing strategy, inventory management, and customer relationship management), and the like, and can be applied to more fields with data analysis requirements, which are not specifically listed herein.

[0053] Optionally, the client 100 can be a separate client specially configured to implement data analysis functions, or a data analysis function module or plug-in in a comprehensive software, such as a data analysis and query module in data mining software, a financial analysis assistant, a data query assistant, and the like in an office automation (OA) client, and is not specifically limited herein.

[0054] Optionally, the client 100 can also be a client of a cloud platform, such as a console of a cloud platform, which can be a web-based console or an application programming interface (API)-based console, and is not specifically limited herein. The console can provide a data analysis type cloud service for a management user, and the user can obtain the use right of the data analysis system 200 provided by the present application by purchasing the cloud service.

[0055] The data processing system 200 and the storage system 300 can be deployed in a computing device or a computing device cluster, where the computing device includes a bare metal server (BMS), a virtual machine, a container or an edge computing device. The BMS refers to a general-purpose physical server, for example, an ARM server or an X86 server; the virtual machine refers to a complete computer system that is simulated by software, runs in a completely isolated environment, and has complete hardware system functions. The work that can be completed in a physical computer can be realized in a virtual machine. When creating a virtual machine in a computing device, part of the hard disk and memory capacity of the physical machine needs to be used as the hard disk and memory capacity of the virtual machine. Each virtual machine has an independent basic input / output system (BIOS), hard disk and operating system, and can be operated like a physical machine; the container is a portable software unit that can combine an application and all its dependencies into a software package that is not limited by the underlying host operating system, so that the complex environment no longer needs to be built, and the application development to deployment process is simplified; the edge computing device refers to a device that is closer to the data source and the end user, has low delay and high bandwidth characteristics, such as a smart router, an edge server and the like. The computing device cluster can include a plurality of the above computing devices, such as a data center, and the present application does not make specific limitations. The storage system 300 can also be deployed in a storage array, such as a redundant array of independent disks (RAID), a network-attached storage (NAS), a storage area network (SAN) and the like, and the present application does not make specific limitations.

[0056] Optionally, the data processing system 200 and the storage system 300 can be deployed in the same computing device, or in different computing devices of the same computing device cluster, or in the computing device and the storage array of the same computing device cluster, or in different computing device clusters, and the present application does not make specific limitations.

[0057] Optionally, the data processing system 200 and the client 100 can be deployed in the same or different computing device clusters, for example, the client 100 is deployed on a terminal device in a first computing device cluster, and the data processing system 200 is deployed on a computing device in a second computing device cluster; or the client 100 and the data processing system 200 are deployed in the same computing device cluster, and it should be understood that the above examples are used for illustration, and the present application does not make specific limitations.

[0058] Further, the data processing system 200 shown in FIG. 1 includes a data acquisition module 210, a data generation module 220, and a data fusion module 230. The data acquisition module 210 is configured to acquire data and corresponding labels from a database, and determine corresponding expert rules according to the labels, and record the correspondence between the data, the labels, and the expert rules. The data generation module 220 is configured to, in a case where the number of data sets corresponding to a label is less than a first threshold, input prompt information into an algorithm model to generate new data corresponding to the label; the prompt information includes one or more of the label, the data, and the expert rule. The data fusion module 230 is configured to fuse a small sample data set and expansion data obtained based on expansion of the small sample data set to obtain a first data set; the data fusion module 230 is further configured to obtain a balanced sample data set according to the first data set and a second data set; the second data set is obtained by performing preprocessing on a large sample data set.

[0059] Next, the data processing method provided by the present application will be further described in combination with the accompanying drawings. Steps S110 to S140 are used to describe the process of obtaining a first data set, step S150 is used to describe the process of obtaining a second data set, and step S160 is used to describe the process of generating a balanced sample data set by fusing the first sample set and the second data set.

[0060] As described above in the system architecture description of the output processing, the data processing includes a small sample data set processing process and a large sample data set processing process. The small sample data set is original data to be processed with a data quantity less than a first threshold, and the large sample data set is original data with a data quantity greater than or equal to a second threshold. First, based on the small sample data set, the small sample data set is expanded according to a first label and an expert rule corresponding to the first label, so that expansion data obtained based on expansion of the small sample data set can be obtained. Then, the expansion data and the small sample data set are fused to obtain a first data set. Then, a second data set is obtained by performing preprocessing (such as undersampling) on a large sample data set. Finally, the first data set and the second data set are fused to obtain a balanced sample data set.

[0061] Next, the process of processing the small sample data set by the present application will be further described in combination with FIG. 2, which includes the following steps:

[0062] S110, acquiring original data and corresponding labels.

[0063] Specifically, S110 can be performed by the data acquisition module 210 shown in FIG. 2.

[0064] In an embodiment, the obtained data is information of a commodity, and the label corresponding to the data is classification information of the commodity, which is used to indicate an attribute of the data. For example, the obtained data is "camera-HWT-D2050-10-I-P(3.6mm) 1T 500 million infrared AI barrel camera-HOLOWITS-no", and the label corresponding to the data is "camera". The data is stored in a database, and each data in the database corresponds to a specific label information. Therefore, by inputting the label "camera" in the commodity database, one or more data corresponding to the label "camera" can be obtained, and the one or more data form a data set.

[0065] The original data and the label are obtained, and after the data is obtained, the outliers in the label are removed. For example, the data is obtained from a database, and a specific obtaining method can be used to input a label to generate data corresponding to the input label. For example, the label "camera" is input, and "assembly component-C32-C322Y-ABCDE-C322Y-ABCDE 200 million infrared zoom hemispherical camera", "new assembly component", and "blank" are queried in the database. The database is composed of data and labels having a mapping relationship with the data. Then, the data is removed. The removal processing includes removal of outliers. The definition method of the outliers can be manual review or intelligent review. For example, according to the experience of experts, "new assembly component" does not belong to the category of the label "camera", and therefore, "new assembly component" can be considered as an outlier and removed. For another example, blank data in the data can also be removed. For a specific removal method, an outlier calculation algorithm can be used for removal, or a loss function can be used to calculate the similarity between the data and a standard value, and whether the data is an outlier is determined according to the similarity. If yes, the outlier is removed. Through the processing of the outliers, the data is clean and accurate.

[0066] As a possible implementation manner, in order to refine the management of the data, different levels of label identification data can also be distinguished.

[0067] Specifically, the label includes a first-level label, a second-level label, and an Nth-level label.

[0068] Optionally, one data set corresponds to one or more labels, and the labels to which the data set belongs can be determined according to the attributes of the data included in the data set.

[0069] Optionally, in specific implementation, multiple labels can also be associated according to the subdivision dimensions of data attributes, that is, multiple labels of the same category and different dimensions are associated with the same data set. For example, the first-level label corresponding to the data "Camera-HWT-D2050-10-I-P(3.6mm) 1T 50 million infrared AI barrel camera-HOLOWITS-no" is "machine vision", the second-level label is "camera", and the third-level label is "D series".

[0070] As a possible implementation, in addition to distinguishing labels into series degrees according to the attributes of data in the data set, the expert rules can also be adaptively subdivided into different expert rules according to the different processing of different data attributes in specific application scenarios. For example, the labels can be distinguished into N (N is a positive integer), including first-level labels, second-level labels, third-level labels, …, and N-level labels; and the expert rules can also be distinguished into N, including first-level expert rules, second-level expert rules, third-level expert rules, …, and N-level expert rules. The correspondence between the labels and the expert rules can be determined in advance according to the data attributes, for example, the first-level labels correspond to the first-level expert rules, the second-level labels correspond to the second-level expert rules, and the third-level labels correspond to the third-level expert rules; and the N-level expert rules include the first-level expert rules, the second-level expert rules, the third-level expert rules, …, and the N-1-level expert rules (N is a positive integer).

[0071] As an example, if the first-level label corresponding to the data is "machine vision", the first-level expert rule corresponding to the first-level label is "text containing camera, video, and machine vision in the description"; if the second-level label corresponding to the data is "camera", the second-level expert rule corresponding to the second-level label is "text containing camera, video, and machine vision in the description" and "text containing camera in the description"; and if the third-level label corresponding to the data is "D series", the third-level expert rule corresponding to the third-level label is "text containing camera, video, and machine vision in the description" and "text containing camera in the description" and "text containing D series in the description". The expert rule in this application refers to the field expert rule, which is a text. Specifically, the expert rule refers to a text corresponding to the label and used to describe the data features under a certain label category. The text usually has professional knowledge, expert description, etc. of the field to which the data belongs. As can be seen, the expert rule is a text, for example, the expert rule is a literal description of the data features. The expression form of the expert rule can be a sentence, a text, and can be English, German, Chinese, without limitation on the language type.

[0072] S120, acquiring the original data and the expert rule corresponding to the label of the original data.

[0073] Specifically, the acquired data is information of a commodity, the label corresponding to the data is classification information of the commodity, the expert rule is a further rule description on the classification information of the commodity combined with an expert knowledge system, and the rule description can be a text representation form including at least one of a term associated with the data, a constraint condition, a data format, and a logical relationship. For example, the acquired data is "camera-HWT-D2050-10-I-P(3.6mm)1T 500 million infrared AI barrel camera-HOLOWITS-no", and the label corresponding to the data is "camera", and the expert rule corresponding to the label is "description contains any of camera and camera". For another example, the label "master node" can correspond to the expert rule "description contains master device standard configuration XXX".

[0074] As an example, the acquired original data "camera-HWT-D2050-10-I-P(3.6mm)1T 500 million infrared AI barrel camera-HOLOWITS-no" and the label corresponding to the original data "camera" and the expert rule corresponding to the label "description contains camera" form a prompt information (prompt template). The label, the data, and the expert rule correspond to each other, or in other words, the label, the data, and the expert rule have a mapping relationship. The label and the data are stored in a database, in other words, the database stores the data, and the data has a corresponding label.

[0075] Optionally, the mapping relationship between the data, the label, and the expert rule can also be extended to one label corresponding to one expert rule, or one label corresponding to one expert rule set. The expert rule set includes a plurality of expert rules. The expert rule set is stored in an expert rule database, and the expert rule database records the expert rule, the expert rule set, and the logical relationship between the expert rules. The logical relationship includes "AND", "OR", and "NOT", and the logical relationship between the expert rules includes any one or a combination of "AND", "OR", and "NOT". For example, "AND" can be understood as an "and" relationship, "OR" can be understood as an "or" relationship, and "NOT" can be understood as a "not including" relationship. For example, when one label corresponds to one expert rule, the expert rule corresponding to the label "camera" is "there is a camera or a camera in the description". When one label corresponds to one expert rule set, the expert rule set corresponding to the label "C-class camera" is "there is a camera or a camera or a camera in the description" AND "the description includes C series", or the expert rule corresponding to the label "software" is "the description contains software" NOT "the description contains a camera". Among them, "AND" and "NOT" are the logical relationships of the expert rules in the expert rule library. For example, "AND" is an "and" logical relationship; "NOT" is a "not including" logical relationship. In other words, when one label corresponds to one expert rule, label A corresponds to expert rule A; when one label corresponds to one expert rule set, label A corresponds to expert rule set B, and expert rule set B is "expert rule A AND expert rule B"; for example, in the case of one label corresponding to one expert rule set, label A corresponds to expert rule set C, and the expert rule set C is "expert rule A NOT expert rule C".

[0076] In S130, the first label and the expert rule corresponding to the first label are input into an algorithm model to generate expanded data obtained based on a small sample data set. The small sample data set is a data set in the original data that belongs to the first label, and the first label is one of the labels.

[0077] In an embodiment, the small sample data set is original data to be processed with a data quantity less than a first threshold, the first label is a label associated with the small sample data set, and the first label is used to indicate an attribute of the data. Before the first label and the expert rule corresponding to the first label are input into the algorithm model, the data acquisition module 210 determines data with a data quantity less than the first threshold from the original data as the small sample data set, and the small sample data set needs to be expanded. At this time, the data acquisition module 210 sends a data input request to the data generation module 220, and the data input request carries the small sample data set, the first label corresponding to the small sample data set, and the expert rule corresponding to the first label. After receiving the request, the data generation module 220 generates expanded data obtained by expanding the small sample data set. For example, the original data includes multiple labels, which are "camera", "node", and "vehicle", respectively. The quantity of original data under the "camera" label is less than 100. Therefore, the data set under the "camera" label is a small sample data set, and the expert rule corresponding to the "camera" label is "one of camera and camera is contained in the description". The label "camera" and the expert rule corresponding to the label "one of camera and camera is contained in the description" are input into the algorithm model to generate expanded data obtained by expanding the small sample data set. For example, the expanded data can be "camera 1T200 million infrared AI barrel camera". The expanded data can be one, but is usually multiple.

[0078] As a possible implementation, the first threshold can be 50, 100, 150, 200, and for example, the range of the first threshold is 50-200. In a specific implementation process, the first threshold can be pre-set according to expert experience, can be determined according to statistical values of historical data, or can be determined according to the quantity proportion of different classification data sets included in the data processing system.

[0079] As a possible implementation, the first label and the expert rule corresponding to the first label are taken as the first prompt information, and the first data set is determined according to the first prompt information; the first prompt information belongs to prompt information, the prompt information includes any one or more of data, expert rules and labels, and the data, expert rules and labels in the prompt information have a mutual mapping relationship. For example, the prompt information is input into an algorithm model to generate new data corresponding to the label. For example, the prompt information composed of the data "camera-HWT-D2050-10-I-P(3.6mm)1T 500 million infrared AI barrel camera-HOLOWITS-no", the label "camera", and the expert rule "description contains camera" is input into an algorithm model to generate new data corresponding to the label "camera-HWM(4.0mm)1T 800 million camera-Peugeot". The algorithm model here can be a data generation module / device such as a generation model or a large language model; the generation process can be performed by an algorithm model, for example, the algorithm model is a generative AI model.

[0080] In a possible implementation, in order to facilitate the evaluation of the quality of the augmented data, the above-mentioned quality evaluation process of the augmented data can also be referred to as data review.

[0081] Specifically, the quality of the augmented data can be evaluated according to a preset manner, for example, a manual review manner or an automated review manner by a data processing system, wherein the automated review manner can adopt any of the following manners:

[0082] Manner 1, the augmented data is reviewed according to the similarity between the augmented data and the label. The calculation method of the similarity includes but is not limited to calculating by a ternary loss function or a cosine similarity. For example, the specific process of calculating by a cosine similarity includes evaluating the similarity between texts by calculating the cosine similarity of the vector representation of the word frequency in the text. First, a vocabulary table is constructed, then the frequency of each word in each text is counted to form a vector, and finally the cosine similarity between these vectors is calculated. Other calculation methods include: using a pre-trained word embedding model (such as Word2Vec, GloVe) to convert words into vectors, then calculating the average value or a specific operation (such as weighted average) of these vectors to obtain the representation of the text, and finally calculating the similarity between the text representations.

[0083] Manner 2, the augmented data is reviewed according to the distance between the augmented data and the label. The calculation method of the distance includes but is not limited to calculating the Euclidean distance, Mahalanobis distance, and Minkowski distance.

[0084] The third mode is to determine whether the text corresponding to the label and / or the expert rule in the expanded data coincides with the label and / or the expert rule, and to review the expanded data. If there is a coincidence, the review is passed; if there is no coincidence, the review is failed.

[0085] S140, fuse the small sample data set and the expanded data obtained based on the small sample data set to obtain a first data set.

[0086] The first data set includes the small sample data set and the expanded data obtained based on the small sample data set. The number of expanded data can be set to an upper threshold value, rather than unlimited expansion. For example, in the case where the number of expanded data reaches a third threshold value, further generation of expanded data is stopped. The third threshold value can be 300, 400, 500, 1000, and the value range of the third threshold value can be 300-1000.

[0087] By setting the first threshold value and the second threshold value, the number of originally obtained data and newly generated data is controllable, and the number of the first sample set is controllable, thereby avoiding the problem of data overfitting, ensuring the diversity of data input into the algorithm model while ensuring the balance and effectiveness of the data, avoiding the tedious adjustment process of hyperparameters and parameters caused by the generation of data by other existing methods such as resampling, simplifying the use of the algorithm model, and improving the working efficiency of the algorithm model by obtaining accurate, effective and controllable data.

[0088] The data expansion of the small sample data in the present application is beneficial to data that lacks or has a small amount of data, and can solve the technical problem of sample imbalance in a sample imbalance processing scenario. The machine learning or algorithm model is prevented from excessively favoring the category of large sample data. In addition, new data is generated by adding the expert rule to the prompt information, so that all generated data accurately match the label and follow the provided expert rule to ensure the consistency and relevance of the data, ensure the controllable number and updateable and adjustable quality of the data, avoid the overfitting problem caused by oversampling of samples in the prior art, reduce the tedious operation of adjusting the parameters and hyperparameters of the model due to sample adjustment, ensure the diversity and accuracy of the data set, and improve the generation and recognition effect of the algorithm model through the above data processing method.

[0089] The above describes the acquisition process of the first sample set. Next, the processing process of the large sample data set is described to obtain a second data set.

[0090] S150, obtaining a large sample data set, and performing preprocessing on the large sample data set. The data amount in the large sample data set is greater than or equal to a second threshold value.

[0091] The second data set is determined as a large sample data set in one embodiment when the data quantity of the data set is greater than a second threshold value; the second threshold value is 2000, 3000, 4000, etc., and the third threshold value is greater than 2000.

[0092] The preprocessing performed on the large sample data set includes undersampling (or upsampling) until the data quantity of the large sample data set reaches a fourth threshold value; and the data set after the undersampling and reaching the fourth threshold value is taken as the second data set.

[0093] In one embodiment, the fourth threshold value is the same as the second threshold value.

[0094] It is worth noting that the undersampling method is not limited in the present application, and as a possible embodiment, any one of random undersampling, Tomek links, One-Sided Selection or a combination of various methods can be used to implement the undersampling processing of data.

[0095] By undersampling the large sample data set, the purpose of balancing the data types is achieved. Different undersampling strategies can be used to sample the most representative data. The sampled data is stored, and a copy of the original data set is retained to facilitate subsequent analysis and reproduction of results. In order to achieve precise control of the balance of data types, the data quantity of the large sample data set after undersampling is precisely controlled and limited. By setting the fourth threshold value to be the same as the second threshold value, the balance of the data after processing of the large sample data set and the small sample data set is precisely guaranteed, the data quantity of the first sample set and the second data set is similar or the same, and the balance of multi-label and multi-type data is achieved.

[0096] As described above, the small sample data set forms the first sample set after processing, and the large sample data set forms the second data set after processing. Next, the process of forming a balanced sample data set after processing of the first sample set and the second data set will be further described.

[0097] S160, obtaining a balanced sample data set according to the first data set and the second data set.

[0098] Specifically, the first sample set has a first sampling weight; and the second data set has a second sampling weight. According to the first sample set and the corresponding first sampling weight, the second data set and the corresponding second sampling weight, a balanced sample data set is determined. For example, the number of the first sample set is 1000, the corresponding first sampling weight is 0.8, the number of the second data set is 1000, and the corresponding second sampling weight is 0.6. Then, the process of forming the balanced sample data set from the first sample set and the second data set is as follows: 800 data (1000*0.8) are randomly selected from the 1000 first samples, and 600 data (1000*0.6) are randomly selected from the 1000 second data sets, and the 1400 data (600+800) form the balanced sample data set.

[0099] Optionally, the first sampling weight and the second sampling weight can be updated according to the data review result. For example, the first sampling weight is adjusted from 0.6 to 0.8, and the second sampling weight is adjusted from 0.5 to 0.3.

[0100] It is worth noting that the initial values of the first sampling weight and the second sampling weight can be configured according to business requirements, and then dynamically updated based on the data review result in the data processing process.

[0101] By setting the sampling weights of different sample sets, on the basis of realizing data balance control, the sampling weights can be dynamically and real-timely adjusted according to the generation effect of the data input algorithm model, so as to realize rapid adjustment of data and improve the processing quality and efficiency of the algorithm model on data.

[0102] In order to more effectively expand the small sample data set, the application also provides an expert rule updating method. Referring to FIG. 3, FIG. 3 is a flowchart of a dynamic updating method for expert rules. As shown in the figure, the updating step specifically includes: first, forming prompt information by combining the expert rules corresponding to each label and the data set, wherein the same label can be associated with one or more expert rules; second, inputting the prompt information into the algorithm model to generate expansion data (which can also be called new data) corresponding to the label; third, reviewing the new data; and fourth, updating the expert rules corresponding to the label according to the review result to obtain updated expert rules.

[0103] Through continuous optimization of the expert rules by editing and updating operations, the description of the expert rules for the label is more accurate, and the quality of the generated data is improved. At the same time, in this process, different expert rules are tested, the expert rules are continuously updated and adjusted, and then tested and / or reviewed. According to the review result, the effectiveness and usability of the generated data are dynamically monitored, the prompt information that can be dynamically adjusted is generated, the quality of the generated data is improved through dynamic optimization of the prompt information, and thus the small sample data with small quantity is reasonably and effectively expanded.

[0104] The method of data processing according to the present application is described in detail above in combination with FIG. 2 to FIG. 3. The data processing system and computing device according to the present application are described below in combination with FIG. 1 and FIG. 4.

[0105] The present application also provides a data processing system 200, as shown in FIG. 1, which comprises:

[0106] The data acquisition module 210 is configured to acquire data and corresponding labels in a database, wherein the labels have corresponding expert rules; acquire a small sample data set and a first label from the data, wherein the small sample data set belongs to a data set, the first label belongs to a label, the amount of data in the small sample data set is less than a first threshold value of raw data to be processed, the first label is a label associated with the small sample data set, and the first label is used to indicate an attribute of data; acquire a large sample data set and perform preprocessing on the large sample data set to obtain a second data set, wherein the amount of data in the large sample data set is greater than or equal to a second threshold value.

[0107] The data generation module 220 is configured to determine a first data set according to the first label and the expert rule corresponding to the first label, wherein the first data set comprises the small sample data set and expansion data obtained based on expansion of the small sample data set, and the expert rule comprises a rule description of an expert experience knowledge system in a specific field associated with the first label, and the rule description comprises a text representation of at least one of a term associated with data, a constraint condition, a data format, and a logical relationship.

[0108] The data fusion module 230 is configured to obtain balanced sample data according to the first data set and the second data set.

[0109] The data acquisition module 210, the data generation module 220, and the data fusion module 230 can be implemented by software or by hardware. For example, the implementation of the A module is described below. Similarly, the implementation of the data acquisition module and the data generation module can refer to the implementation of the data fusion module.

[0110] As an example of a software functional unit, the data acquisition module can include code running on a compute instance. The compute instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the compute instance can be one or more. For example, module A can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers running the code can be distributed in the same region, or in different regions. Further, the multiple hosts / virtual machines / containers running the code can be distributed in the same availability zone (AZ), or in different AZs, each of which includes one data center or multiple data centers in close geographical proximity. Generally, a region can include multiple AZs.

[0111] Similarly, the multiple hosts / virtual machines / containers running the code can be distributed in the same virtual private cloud (VPC), or in multiple VPCs. Generally, a VPC is set up within a region, and a communication gateway needs to be set up in each VPC for cross-region communication between two VPCs in the same region or between VPCs in different regions, and the interconnection between the VPCs is realized through the communication gateway.

[0112] As an example of a hardware functional unit, the data acquisition module can include at least one computing device, such as a server, etc. Alternatively, the data acquisition module can also be a device implemented using a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or a programmable logic device (PLD), etc. The PLD can be implemented by a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system on chip (SoC), an offload card, an acceleration card, or any combination thereof.

[0113] The multiple computing devices included in the data acquisition module can be distributed in the same region or in different regions. The multiple computing devices included in the data acquisition module can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the data acquisition module can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offload cards, accelerator cards, and other computing devices.

[0114] The data processing system 200 described in the present application may correspond to executing the method described in the present application, and the above-mentioned and other operations and / or functions of each unit in the data processing system 200 are respectively for implementing the corresponding processes of each method in Figures 2 to 3. For the sake of brevity, they will not be repeated here.

[0115] As a possible implementation, the data acquisition module 210 is further configured to acquire raw data sets, labels, and expert rules. The raw data sets include multiple data sets with different data attributes, the label library includes labels indicating different data attributes, and the expert rule library includes multiple expert rules and logical relationships between the multiple expert rules. For example, the data acquisition module 210 includes a database and an expert rule library. The database is configured to store data sets, and the data in the data sets have corresponding labels. For example, "Camera M3857" has the label "camera." The data label represents the data attributes, such as the data classification information. The expert rule library includes multiple expert rules. After the data acquisition module acquires the data and its labels, it can further acquire the expert rule matching the data set from the expert rule library, such as "the descriptions of camera, video camera, etc. appear and are applied to ticket acquisition." It can also acquire multiple expert rules matching the data set and the logical relationships between the multiple expert rules from the expert rule library. For example, "the descriptions of camera, video camera, etc. appear" NOT "applied to the aviation field." Here, NOT represents a logical relationship, which can be understood as meaning "not included." The data acquisition module 210 is further configured to construct a mapping relationship between data, labels, and expert rules based on data attributes. The mapping relationship between data, labels, and expert rules indicates the mapping relationship between the at least one label, the multiple expert rules, and the multiple data sets. For example, the label for "Camera M3857" is "camera," and its corresponding expert rule is "the descriptions such as camera, video camera, etc. appear" NOT "apply to the flight field."

[0116] As a possible implementation manner, the data acquisition module is further configured to determine, according to the first label, an expert rule corresponding to the first label from a mapping relationship between the data labels and the expert rules; the data generation module is configured to use the large language model to generate the augmented data according to the first label and the expert rule corresponding to the first label as first prompt information; and when the number of the augmented data reaches a third threshold, the augmented data and the small sample data set are fused to obtain the first data set. For example, the first label corresponding to the small sample data set is "camera", and the data set, the label and the expert rule are obtained in the data acquisition module. For example, one of the data is "camera M3857", which has the label "camera", and the corresponding expert rule is "appear camera, camera, etc. description" NOT "applied to the field of flight". The label "camera M3857" has the label "camera", and the corresponding expert rule is "appear camera, camera, etc. description" NOT "applied to the field of flight". The new generated augmented data "camera X6599" can be obtained by inputting the label "camera M3857" into the large model. The generation of the augmented data is stopped when the number of the augmented data reaches the third threshold.

[0117] As a possible implementation manner, the data processing module is configured to perform undersampling processing on the large sample data set to obtain a processing result; and when the number of data of the processing result reaches a fourth threshold, the data set that has undergone the undersampling processing and has the number of data reaching the fourth threshold is taken as the second data set.

[0118] As a possible implementation method, the data fusion module is configured to respectively acquire a first sampling weight corresponding to the first data set and a second sampling weight corresponding to the second data set; and determine a balanced sample data set according to the first data set and the first sampling weight corresponding to the first data set, the second data set and the second sampling weight corresponding to the second data set.

[0119] The application further provides a chip system, which comprises a processor and a power supply circuit. The power supply circuit is configured to supply power to the processor. The processor is configured to execute a corresponding process of each method in FIG. 2 to FIG. 3. For brevity, details are not repeated here. The processor can be implemented by a GPU, or can be implemented by a DPU, NPU, XPU, SoC, offload card, acceleration card or other computing device.

[0120] The application further provides a computing device 100. As shown in FIG. 4, the computing device 100 comprises a bus 102, a processor 104, a memory 106 and a communication interface 108. The processor 104, the memory 106 and the communication interface 108 communicate through the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that the number of processors and memories in the computing device 100 is not limited in the application.

[0121] Bus 102 can be a peripheral component interconnect Express (PCIe) bus or an extended industry standard architecture (EISA) bus, a unified bus (UBus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), or the like. The unified bus can also be referred to as a coherent bus. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one line is shown in FIG. 4, but it does not mean that there is only one bus or only one type of bus. Bus 104 can include a path for transmitting information between various components of computing device 100 (e.g., memory 106, processor 104, communication interface 108). The unified bus can also be referred to as a coherent bus.

[0122] Processor 104 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), an ASIC, a FPGA, a CPLD, an NPU, a SoC, an offload card, an acceleration card, or the like computing device.

[0123] Memory 106 can include volatile memory, such as random access memory (RAM). Processor 104 can also include non-volatile memory, such as read-only memory (ROM), a floppy disk, a compact disk, an optical disk, a hard disk, a solid state drive, or the like. Additionally, memory 106 can be implemented using a storage class memory (SCM), a phase change memory (PCM), or other types of storage media.

[0124] It is worth mentioning that the same type of storage medium can be configured in the same computing device to implement the memory 106 function, and two or more types of storage media can also be configured to implement the memory 106 function, and the present application does not limit this.

[0125] The memory 106 stores executable program code, and the processor 104 executes the executable program code to respectively implement the functions of the aforementioned A module, B module and C module, thereby implementing the corresponding flow of each method in FIGS. 2-3. That is, the memory 106 has instructions for executing the corresponding flow of each method in FIGS. 2-3.

[0126] The communication interface 103 uses a transceiving module such as but not limited to a network interface card, a transceiver, etc. to implement communication between the computing device 100 and other devices or communication networks.

[0127] As a possible implementation manner, the computing device 100 can also include a chip system including a processor and a power supply circuit for performing power supply to the processor, and the processor is used to execute the operation steps corresponding to the corresponding flow of each method in FIGS. 2-3. For the sake of brevity, it will not be repeated here. Among them, the processor can be implemented by GPU, and can also be implemented by DPU, NPU, XPU, SoC, offload card, acceleration card and other computing devices or AI chips.

[0128] As a possible implementation manner, the computing device 100 can include multiple types of processors 104, that is, the computing device 100 is a heterogeneous device, for example, the computing device 100 includes CPU and GPU, and at least one of the processors 104 in the computing device 100 can execute the operation steps corresponding to the corresponding flow of each method in FIGS. 2-3. For the sake of brevity, it will not be repeated here.

[0129] The above-described embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented by software, the above-described embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing one or more available medium sets. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid state disk (SSD).

[0130] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized by, The method is executed by a data processing system, and comprises: obtaining a small sample data set and a first label, the small sample data set belonging to a data set, the first label belonging to a label, the data amount of the small sample data set being less than a first threshold value of to-be-processed original data, the first label being one label associated with the small sample data set, and the first label being used to indicate one attribute of data; determining a first data set according to the first label and an expert rule corresponding to the first label, the first data set comprising the small sample data set and expansion data obtained based on expansion of the small sample data set, and the expert rule comprising an expert rule description of an expert experience knowledge system in a specific field associated with the first label, the expert rule description comprising a text representation of at least one of a term associated with data, a constraint condition, a data format, and a logical relationship; obtaining a large sample data set, and performing preprocessing on the large sample data set to obtain a second data set, the data amount of the large sample data set being greater than or equal to a second threshold value; obtaining a balanced sample data set according to the first data set and the second data set.

2. The method of claim 1, wherein, Before the small sample data set and the first label are obtained, the method further comprises: obtaining an original data set, a label library, and an expert rule library, wherein the original data set comprises a plurality of data sets having different data attributes, the label library comprises labels used to indicate different data attributes, and the expert rule library comprises a plurality of expert rules and logical relationships between the plurality of expert rules; constructing a mapping relationship of data, labels, and expert rules according to data attributes, the mapping relationship of data, labels, and expert rules being used to indicate a mapping relationship between the at least one label, the plurality of expert rules, and the plurality of data sets.

3. The method of claim 2, wherein, The determining of the first data set according to the first label and the expert rule corresponding to the first label comprises: determining the expert rule corresponding to the first label from the mapping relationship of data, labels, and expert rules according to the first label; using a large language model to generate the expansion data according to the first label and the expert rule corresponding to the first label as first prompt information; when the number of the expansion data reaches a third threshold value, fusing the expansion data and the small sample data set to obtain the first data set.

4. The method of claim 1, wherein, The preprocessing of the large sample data set to obtain the second data set comprises: performing undersampling processing on the large sample data set to obtain a processing result; when the data amount of the processing result reaches a fourth threshold value, taking a data set that has undergone the undersampling processing and has the data amount reaching the fourth threshold value as the second data set.

5. The method according to any one of claims 1 to 4, characterized in that, The obtaining of the balanced sample data set according to the first data set and the second data set comprises: respectively obtaining a first sampling weight corresponding to the first data set and a second sampling weight corresponding to the second data set; determining the balanced sample data set according to the first data set, the first sampling weight corresponding to the first data set, the second data set, and the second sampling weight corresponding to the second data set.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: The review result of the augmented data is determined according to a preset mode, and the preset mode includes any one of a similarity between the augmented data and the first label or a distance between the augmented data and the first label; The expert rule corresponding to the label and the prompt information are updated according to the review result.

7. A data processing system, characterized by The data processing system comprises: The data acquisition module is configured to acquire data and corresponding labels in a database, wherein the labels have corresponding expert rules, acquire a small sample data set and a first label from the data, the small sample data set belongs to a data set, the first label belongs to a label, the amount of data in the small sample data set is less than a first threshold value, the first label is a label associated with the small sample data set, and the first label is used to indicate an attribute of data, acquire a large sample data set, and perform preprocessing on the large sample data set to obtain a second data set, wherein the large sample data set is a data set stored in the data processing system, and the amount of data in the large sample data set is greater than or equal to a second threshold value. The data generation module is configured to determine a first data set according to the first label and an expert rule corresponding to the first label, wherein the first data set comprises the small sample data set and augmented data obtained by augmenting the small sample data set, and the expert rule comprises an expert rule description of an expert experience knowledge system in a specific field associated with the first label, and the expert rule description comprises a text representation of at least one of a term associated with data, a constraint condition, a data format, and a logical relationship. The data fusion module is configured to obtain a balanced sample data set according to the first data set and the second data set.

8. The data processing system of claim 7, wherein The data acquisition module is further configured to acquire an original data set, a label library, and an expert rule library, wherein the original data set comprises a plurality of data sets having different data attributes, the label library comprises labels used to indicate different data attributes, and the expert rule library comprises a plurality of expert rules and logical relationships between the plurality of expert rules. The data acquisition module is further configured to construct a mapping relationship among data, labels, and expert rules according to data attributes, and the mapping relationship among the data, the labels, and the expert rules is used to indicate a mapping relationship between the at least one label, the plurality of expert rules, and the plurality of data sets.

9. The data processing system of claim 8, wherein, The data acquisition module is further configured to determine an expert rule corresponding to the first label from the mapping relationship among the data, the labels, and the expert rules according to the first label. The data generation module is configured to use a large language model to generate the augmented data according to the first label and the expert rule corresponding to the first label as first prompt information. When the amount of the augmented data reaches a third threshold value, the augmented data and the small sample data set are fused to obtain the first data set.

10. The data processing system of claim 7, wherein The data processing module is configured to perform under-sampling processing on the large sample data set to obtain a processing result, and when the data quantity of the processing result reaches a fourth threshold, a data set that has been processed by under-sampling and has a data quantity reaching the fourth threshold is taken as the second data set.

11. The data processing system of claim 7, wherein, The data fusion module is configured to obtain a first sampling weight corresponding to the first data set and a second sampling weight corresponding to the second data set, respectively, and determine a balanced sample data set according to the first data set and the first sampling weight corresponding to the first data set, the second data set and the second sampling weight corresponding to the second data set.

12. A chip system, characterized by The chip system comprises a processor and a power supply circuit, the power supply circuit is configured to supply power to the processor, and the processor is configured to execute the operation steps of the method according to any one of claims 1 to 6.

13. A computer device, comprising: The computing device comprises a processor and a memory, the processor and the memory are connected through a bus and complete communication between each other, the memory is configured to store computer execution instructions, and the processor is configured to execute the computer execution instructions in the memory to execute the operation steps of the method according to any one of claims 1 to 6 by using hardware resources in the computing device when the computing device is running.

14. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the operation steps of the method according to any one of claims 1 to 6.

15. A computer program product, characterised in that, The computer program product stores computer readable instructions, and the computer readable instructions are executed by the processor to realize the operation steps of the method according to any one of claims 1 to 6.

Images