Abnormal data processing method, device, electronic device and storage medium

Abstract

The present application relates to the field of data processing technology, and provides an abnormal data processing method, device, electronic device and storage medium. In the present application, the first data to be processed is input into a pre-trained codec to obtain the second data output by the codec; the minimum value in the second data is used as the lower limit of the target interval, and the maximum value in the second data is used as the upper limit of the target interval, so as to determine the target interval; the values ​​in the first data that are outside the target interval are determined as abnormal data, and the abnormal data is cleared. Compared with the prior art that clears abnormal data based on human experience, the present application uses the codec to determine the target interval, so as to automatically identify and clear abnormal data based on the target interval, and has a higher efficiency in clearing abnormal data.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an abnormal data processing method, device, electronic device and storage medium. Background Art

[0002] With the development of IoT technology, more and more IoT devices are being applied to large-scale agricultural production. These IoT devices usually include humidity sensors, temperature sensors, pH sensors, light intensity sensors, carbon dioxide sensors, water and fertilizer control systems, etc. The data collected by these IoT devices can be used to predict or identify crop pests, crop diseases, crop growth, crop growth period, etc. However, since these IoT devices are usually installed directly outdoors, the accuracy of their data collection will be affected when facing extreme environments (such as flooding). Therefore, when using IoT devices for prediction (such as pest prediction), it is necessary to clear outliers in the data in advance, and then use clean data for prediction to ensure the accuracy of the prediction results.

[0003] However, in the prior art, abnormal data is usually cleared based on human experience. However, the efficiency of this abnormal data clearing method is low, and there may be problems of missed clearing or over-clearing, which in turn affects the accuracy of the prediction results. Summary of the invention

[0004] The present application provides an abnormal data processing method, device, electronic device and storage medium, aiming to improve the efficiency of clearing abnormal data.

[0005] A first aspect of an embodiment of the present application provides a method for processing abnormal data, the method comprising:

[0006] Inputting first data to be processed into a pre-trained codec to obtain second data output by the codec; the first data and the second data both include multiple values;

[0007] The target interval is determined by taking the minimum value of the second data as the lower limit of the target interval and taking the maximum value of the second data as the upper limit of the target interval;

[0008] The values ​​of the first data that are outside the target interval are determined as abnormal data, and the abnormal data are cleared.

[0009] Optionally, the method further comprises: training the codec;

[0010] The training of the codec comprises:

[0011] Inputting sample data into the codec to obtain a first output result of the codec; the sample data includes a plurality of sample values, wherein the plurality of sample values ​​include a plurality of normal values ​​and one or more abnormal values;

[0012] Inputting the first output result into a pre-trained classification model to obtain a predicted classification result output by the classification model;

[0013] Determine a first loss value according to the first output result and normal values ​​and abnormal values ​​in the sample data;

[0014] Determining a second loss value according to the predicted classification result and the actual classification result corresponding to the sample data;

[0015] The codec is updated according to the first loss value and the second loss value.

[0016] Optionally, determining a first loss value according to the first output result and normal values ​​and abnormal values ​​in the sample data includes:

[0017] Determine the first loss value according to the magnitude relationship between the maximum value in the first output result and the maximum normal value in the sample data, the magnitude relationship between the maximum value in the first output result and the first abnormal value in the sample data, and / or the magnitude relationship between the minimum value in the first output result and the minimum normal value in the sample data, and the magnitude relationship between the minimum value in the first output result and the second abnormal value in the sample data;

[0018] The first abnormal value refers to the smallest abnormal value among the abnormal values ​​that are greater than the maximum normal value, and the second abnormal value refers to the largest abnormal value among the abnormal values ​​that are less than the minimum normal value.

[0019] Optionally, the predicted classification result and the actual classification result are both in vector form; and determining the second loss value according to the predicted classification result and the actual classification result corresponding to the sample data includes:

[0020] Calculating a vector distance between the predicted classification result and the actual classification result;

[0021] The second loss value is determined according to the vector distance.

[0022] Optionally, updating the codec according to the first loss value and the second loss value includes:

[0023] determining a third loss value according to a first weight corresponding to the first loss value, the first loss value, a second weight corresponding to the second loss value, and the second loss value;

[0024] updating the codec using the third loss value;

[0025] The first weight and / or the second weight is determined based on the predicted classification results and the actual classification results in the most recent N trainings.

[0026] Optionally, there are multiple classification models, and each classification model performs different classification tasks; determining the second loss value according to the predicted classification result and the actual classification result corresponding to the sample data includes:

[0027] The second loss value is determined according to the predicted classification result of each classification model and the actual classification result corresponding to the sample data under the corresponding classification task.

[0028] Optionally, the multiple classification models include at least two of the following: a crop pest classification model, a crop disease identification model, a crop growth identification model, and a crop growth period identification model.

[0029] A second aspect of an embodiment of the present application provides an abnormal data processing device, the device comprising:

[0030] A data input module, used for inputting first data to be processed into a pre-trained codec to obtain second data output by the codec; the first data and the second data both include multiple values;

[0031] an interval determination module, configured to use the minimum value in the second data as the lower limit of the target interval and the maximum value in the second data as the upper limit of the target interval, thereby determining the target interval;

[0032] The abnormal data clearing module is used to determine the values ​​in the first data that are outside the target interval as abnormal data, and clear the abnormal data.

[0033] A third aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor;

[0034] The memory is used to store application programs;

[0035] The processor is used to run the application stored in the memory to implement any of the above-mentioned abnormal data processing methods.

[0036] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein an application is stored in the computer-readable storage medium, and when the application is executed by a processor, it is used to implement any of the above-mentioned abnormal data processing methods.

[0037] The abnormal data processing method provided by the present application is adopted, the first data to be processed is input into a pre-trained codec, and the second data output by the codec is obtained; the minimum value in the second data is used as the lower limit of the target interval, and the maximum value in the second data is used as the upper limit of the target interval, so as to determine the target interval; the values ​​in the first data that are outside the target interval are determined as abnormal data, and the abnormal data is cleared. Compared with the prior art that clears abnormal data based on human experience, the present application uses the codec to determine the target interval, so as to automatically identify and clear abnormal data based on the target interval, and has a higher efficiency in clearing abnormal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0039] Figure 1 It is a flowchart of an abnormal data processing method provided by an embodiment of the present application;

[0040] Figure 2 is a schematic diagram of a process of training a codec provided by an embodiment of the present application;

[0041] Figure 3 This is a schematic diagram of the relationship between the first output result and the sample data provided in an embodiment of the present application. Figure 1 ;

[0042] Figure 4 This is a schematic diagram of the relationship between the first output result and the sample data provided in an embodiment of the present application. Figure 2 ;

[0043] Figure 5 This is a schematic diagram of the relationship between the first output result and the sample data provided in an embodiment of the present application. Figure 3 ;

[0044] Figure 6 is a structural diagram of an abnormal data processing device provided by an embodiment of the present application;

[0045] Figure 7 It is a structural block diagram of an electronic device proposed in one embodiment of the present application.

[0046] In the figure: 410 data input module, 420 interval determination module, 430 abnormal data clearing module, 500 electronic device, 510 processor, 520 memory. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0048] In large-scale agricultural production, the data collected by IoT devices can be used to predict or identify crop pests, crop diseases, crop growth, crop growth period, etc. However, since these IoT devices are usually installed directly outdoors, the accuracy of their data collection will be affected when facing extreme environments (such as flooding). Therefore, when using IoT devices for prediction (such as pest prediction), it is necessary to clear outliers in the data in advance, and then use clean data for prediction to ensure the accuracy of the prediction results. However, in the prior art, abnormal data removal is usually based on human experience. However, the efficiency of this abnormal data removal method is low, and there may be problems of missed removal or over-clearing, which in turn affects the accuracy of the prediction results.

[0049] In order to improve the efficiency of clearing abnormal data, this application provides a method for processing abnormal data. Figure 1 , Figure 1 FIG. 1 is a flow chart of an abnormal data processing method provided by an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0050] S110: Inputting first data to be processed into a pre-trained codec to obtain second data output by the codec; both the first data and the second data include multiple values.

[0051] The first data to be processed may or may not include abnormal values. An abnormal value refers to a value that deviates from the normal range. For example, the first data to be processed is a plurality of temperature values ​​recorded over time. These temperature values ​​are collected by a thermometer set in a greenhouse. One of these temperature values ​​is 72.2°C, which obviously deviates from the normal temperature range and is an abnormal value. Another temperature value among these temperature values ​​is 12°C, which also has a certain deviation from the normal range. However, it is impossible to accurately judge whether this temperature value is an abnormal value through human experience.

[0052] In the present application, after receiving a string of numerical values ​​(i.e., first data), the encoding module in the codec encodes the first data, and then inputs the encoding result into the value decoding module, which decodes the encoding result to output a string of processed numerical values ​​(i.e., second data).

[0053] S120: Taking the minimum value in the second data as the lower limit of the target interval, and taking the maximum value in the second data as the upper limit of the target interval, thereby determining the target interval.

[0054] In the present application, the second data output by the codec includes multiple values, the maximum value among the multiple values ​​is used as the upper limit of the target interval, and the minimum value among the multiple values ​​is used as the lower limit of the target interval, so as to determine the target interval.

[0055] S130: Determine the values ​​of the first data that are outside the target interval as abnormal data, and clear the abnormal data.

[0056] In the present application, the first data includes multiple values. If there are values ​​outside the target interval among the multiple values, these values ​​are determined as abnormal values ​​and the abnormal data are removed. If there is no value outside the target interval among the multiple values, it is determined that the first data does not include abnormal data.

[0057] Compared with the prior art that removes abnormal data based on human experience, the present application uses a codec to determine a target interval, thereby automatically identifying and removing abnormal data based on the target interval, and has a higher efficiency in removing abnormal data.

[0058] In some specific implementations, the abnormal data processing method further includes: training a codec.

[0059] refer to Figure 2 , Figure 2 Schematic diagram of the process of training a codec provided by an embodiment of the present application. Figure 2 As shown, firstly, the sample data is input into the codec to obtain the first output result of the codec. Among them, the sample data includes multiple sample values, and the multiple sample values ​​include multiple normal values ​​and one or more abnormal values. For ease of understanding, illustratively, the first sample data is a string of temperature values, and the normal values ​​among these temperature values ​​are concentrated between 18.2-26.5°C, and these temperature values ​​also include abnormal values ​​such as 4.6°C, 12.6°C, 38.5°C, and 52.7°C. The first output result output by the codec also includes multiple processed temperature values. The second sample data is a string of humidity values, and the normal values ​​among these humidity values ​​are concentrated between 34.5%-80.2%, and these humidity values ​​also include abnormal values ​​such as 0.1%, 1.6%, 99.1%, and 99.9%. The first output result output by the codec also includes multiple processed humidity values.

[0060] like Figure 2As shown, the first output result output by the codec is then input into the pre-trained classification model to obtain the predicted classification result output by the classification model. For ease of understanding, exemplarily, the classification model is a crop disease recognition model. For the first output result output by the decoder, the processed temperature value and humidity value are first cross-arranged to obtain fused data. For example, the first temperature value is used as the first value of the fused data, the first humidity value is used as the second value of the fused data, the second temperature value is used as the third value of the fused data, the second humidity value is used as the fourth value of the fused data... and so on, to obtain fused data. The fused data is then input into the pre-trained crop disease recognition model, so that the crop disease recognition model outputs a predicted classification result based on the fused data, that is, a disease prediction result.

[0061] like Figure 2 As shown, a first loss value is determined based on the first output result and the normal values ​​and abnormal values ​​in the sample data, and a second loss value is determined based on the predicted classification result and the actual classification result corresponding to the sample data, and finally the codec is updated based on the first loss value and the second loss value.

[0062] In this application, when training the codec, a pre-trained classification model is also introduced. Not only the first loss value is determined according to the first output result of the codec and the normal values ​​and abnormal values ​​in the sample data, but also the second loss value is determined according to the predicted classification result and the actual classification result of the classification model, so as to update the codec according to the first loss value and the second loss value at the same time. In this way, the trained codec can not only be used to process abnormal values ​​(that is, accurately predict the interval where the normal value is located), but also further ensure that the clean data after processing the abnormal values ​​can be effectively used for classification prediction, ensuring the accuracy of classification prediction.

[0063] In some specific embodiments, the first loss value is determined based on the first output result and the normal values ​​and abnormal values ​​in the sample data, specifically including: determining the first loss value based on the size relationship between the maximum value in the first output result and the maximum normal value in the sample data, the size relationship between the maximum value in the first output result and the first abnormal value in the sample data, and / or the size relationship between the minimum value in the first output result and the minimum normal value in the sample data, and the size relationship between the minimum value in the first output result and the second abnormal value in the sample data; wherein the first abnormal value refers to the minimum abnormal value among the abnormal values ​​greater than the maximum normal value, and the second abnormal value refers to the maximum abnormal value among the abnormal values ​​less than the minimum normal value.

[0064] Specifically, if Figure 3As shown, if the maximum value of the first output result is greater than the first abnormal value, and / or the minimum value of the first output result is less than the second abnormal value, it means that the codec fails to identify the boundary between the normal value and the abnormal value, the current processing capability of the codec is not ideal, and a larger first loss value needs to be determined, so as to update the codec to a greater extent. Figure 3 In the figure, the solid points are sample data, and the hollow points are the first output results.

[0065] For example, if the maximum value of the first output result is greater than the first abnormal value, then the formula Calculate the first loss value, where loss1 is the first loss value, V max Refers to the maximum normal value in the sample data, V min Refers to the minimum normal value in the sample data, V ′ max refers to the maximum value of the first output result, the function It refers to taking The minimum value of and 1, the function It means taking 0.5 and The maximum of the two.

[0066] If the minimum value of the first output result is smaller than the second abnormal value, then the formula Calculate the first loss value, where V ′ min Refers to the minimum value of the first output result.

[0067] If the maximum value of the first output result is greater than the first abnormal value, and the minimum value of the first output result is less than the second abnormal value, then according to the formula Calculate the first loss value, where

[0068] like Figure 4 As shown, if the maximum value of the first output result is less than the first abnormal value and greater than the maximum normal value in the sample data, and the minimum value of the first output result is greater than the second abnormal value and less than the minimum normal value in the sample data, it means that the current processing capacity of the codec is relatively ideal and does not need to be updated temporarily. Therefore, the first loss value can be determined as 0 at this time. Figure 4 In the figure, the solid points are sample data, and the hollow points are the first output results.

[0069] like Figure 5As shown, if the maximum value of the first output result is smaller than the maximum normal value in the sample data, and / or the minimum value of the first output result is larger than the minimum normal value in the sample data, it means that although the codec can identify abnormal values, it will also mistake normal values ​​for abnormal values. It can be seen that the current processing capability of the codec is still insufficient, and a smaller first loss value needs to be determined so as to update the codec to an appropriate degree. Figure 5 In the figure, the solid points are sample data, and the hollow points are the first output results.

[0070] For example, if the maximum value of the first output result is less than the maximum normal value in the sample data, then the formula Calculate the first loss value.

[0071] If the minimum value of the first output result is greater than the minimum normal value in the sample data, then the formula Calculate the first loss value.

[0072] If the maximum value of the first output result is smaller than the maximum normal value in the sample data, and the minimum value of the first output result is larger than the minimum normal value in the sample data, then the formula Calculate the first loss value, where

[0073] It should be noted that if the sample data has more than one category, for example, the sample data includes first sample data (for example, the first sample data is multiple temperature values) and second sample data (for example, the second sample data is multiple humidity values), then for each category of sample data, the first loss value is calculated according to the above method, and then the average of the first loss values ​​of multiple categories of sample data is calculated, and the average value is used as the final first loss value.

[0074] In some specific implementations, the predicted classification result and the actual classification result are both in vector form; determining the second loss value according to the predicted classification result and the actual classification result corresponding to the sample data includes: calculating the vector distance between the predicted classification result and the actual classification result; and determining the second loss value according to the vector distance. Specifically, if the difference between the predicted classification result and the actual classification result is large, the larger the vector distance is. In some embodiments, the vector distance can be determined as the second loss value.

[0075] For ease of understanding, exemplarily, the classification model is a crop disease recognition model, and the predicted classification result output by the crop disease recognition model is specifically a vector {z1, z2, z3, z4, z5} of length 5, where z1, z2, z3, z4, z5 are all values ​​greater than or equal to 0 and less than or equal to 1, and the sum of z1, z2, z3, z4, z5 is equal to 1, and z1, z2, z3, z4, z5 respectively represent the probability values ​​of different types of crop diseases. Similarly, the actual classification result is also a vector of length 5.

[0076] In some specific embodiments, updating the codec according to the first loss value and the second loss value includes: determining a third loss value according to a first weight corresponding to the first loss value, the first loss value, a second weight corresponding to the second loss value, and the second loss value; updating the codec using the third loss value; wherein the first weight and / or the second weight is determined based on the predicted classification results and the actual classification results in the most recent N training times.

[0077] Specifically, the product of the first weight and the first loss value is added to the product of the second weight and the second loss value to obtain a third loss value, wherein the sum of the first weight and the second weight is equal to 1. In some embodiments, the greater the deviation between the predicted classification results and the actual classification results in the most recent N trainings, the closer the second weight is to 1, so that the codec can give priority to learning the first capability, which means that clean data after processing outliers can be effectively used for classification prediction, thereby ensuring the most basic agricultural production needs. In some embodiments, the smaller the deviation between the predicted classification results and the actual classification results in the most recent N trainings, the closer the second weight is to 0, so that the codec can then learn the second capability, which means accurately predicting the interval where the normal value is located, thereby helping to effectively remove outliers.

[0078] Exemplarily, for each of the most recent N trainings, the vector distance between the predicted classification result and the actual classification result in the training is calculated, and then the average of the N vector distances of the most recent N trainings is calculated, so that the average of the N vector distances is used as the second weight. Accordingly, the first weight is equal to 1 minus the second weight.

[0079] In some specific implementations, there are multiple classification models, and each classification model performs different classification tasks. In the present application, determining the second loss value according to the predicted classification result and the actual classification result corresponding to the sample data includes: determining the second loss value according to the predicted classification result of each classification model and the actual classification result corresponding to the sample data under the corresponding classification task.

[0080] For ease of understanding, illustratively, it is assumed that the classification model includes a crop disease recognition model and a crop growth recognition model, wherein the classification task performed by the crop disease recognition model is to recognize the type of crop disease, and the classification task performed by the crop growth recognition model is to recognize the growth stage of the crop. For a given sample data, when it is applied to the recognition task of "recognizing the type of crop disease", the corresponding actual classification result is a1, and when it is applied to the recognition task of "recognizing the growth stage of the crop", the corresponding actual classification result is b1. In the above training process, the predicted classification result output by the crop disease recognition model is a2, and the predicted classification result output by the crop growth recognition model is b2. Then, a second loss value is calculated based on a1 and a2, and another second loss value is calculated based on b1 and b2, and then the average of the two second loss values ​​is determined as the final second loss value.

[0081] In some specific embodiments, the multiple classification models include at least two of the following: a crop pest classification model, a crop disease identification model, a crop growth identification model, and a crop growth period identification model.

[0082] Based on the same inventive concept, the present application also provides an abnormal data processing device, referring to Figure 6 , Figure 6 Schematic diagram of the structure of an abnormal data processing device provided by an embodiment of the present application. Figure 6 As shown, the device comprises:

[0083] The data input module 410 is used to input the first data to be processed into a pre-trained codec to obtain the second data output by the codec; the first data and the second data both include multiple values;

[0084] An interval determination module 420, configured to use the minimum value in the second data as the lower limit of the target interval and the maximum value in the second data as the upper limit of the target interval, thereby determining the target interval;

[0085] The abnormal data clearing module 430 is used to determine the values ​​in the first data that are outside the target interval as abnormal data, and clear the abnormal data.

[0086] Compared with the prior art that removes abnormal data based on human experience, the present application uses a codec to determine a target interval, thereby automatically identifying and removing abnormal data based on the target interval, and has a higher efficiency in removing abnormal data.

[0087] In some specific embodiments, the device further includes a training module, which specifically includes:

[0088] A first input unit is used to input sample data into the codec to obtain a first output result of the codec; the sample data includes a plurality of sample values, and the plurality of sample values ​​include a plurality of normal values ​​and one or more abnormal values;

[0089] A second input unit is used to input the first output result into a pre-trained classification model to obtain a predicted classification result output by the classification model;

[0090] A first loss value determining unit, configured to determine a first loss value according to the first output result and normal values ​​and abnormal values ​​in the sample data;

[0091] A second loss value determining unit, used to determine a second loss value according to the predicted classification result and the actual classification result corresponding to the sample data;

[0092] An updating unit is used to update the codec according to the first loss value and the second loss value.

[0093] In some specific embodiments, the first loss value determining unit is specifically used to: determine the first loss value according to the size relationship between the maximum value in the first output result and the maximum normal value in the sample data, the size relationship between the maximum value in the first output result and the first abnormal value in the sample data, and / or the size relationship between the minimum value in the first output result and the minimum normal value in the sample data, and the size relationship between the minimum value in the first output result and the second abnormal value in the sample data;

[0094] The first abnormal value refers to the smallest abnormal value among the abnormal values ​​that are greater than the maximum normal value, and the second abnormal value refers to the largest abnormal value among the abnormal values ​​that are less than the minimum normal value.

[0095] In some specific implementations, the predicted classification result and the actual classification result are both in vector form, and the second loss value determination unit is specifically used to: calculate the vector distance between the predicted classification result and the actual classification result; and determine the second loss value based on the vector distance.

[0096] In some specific embodiments, the update unit is specifically used to: determine a third loss value based on a first weight corresponding to the first loss value, the first loss value, a second weight corresponding to the second loss value, and the second loss value; update the codec using the third loss value; wherein the first weight and / or the second weight is determined based on the predicted classification results and actual classification results in the most recent N training times.

[0097] In some specific embodiments, there are multiple classification models, and the classification tasks performed by each classification model are different; the second loss value determination unit is specifically used to determine the second loss value based on the predicted classification results of each classification model and the actual classification results corresponding to the sample data under the corresponding classification task.

[0098] In some specific embodiments, the multiple classification models include at least two of the following: a crop pest classification model, a crop disease identification model, a crop growth identification model, and a crop growth period identification model.

[0099] See also Figure 7 , Figure 7 It is a structural block diagram of an electronic device proposed in an embodiment of the present application. The electronic device 500 includes a processor 510, a memory 520, and one or more applications, wherein the one or more applications are stored in the memory 520 and are configured to be executed by one or more processors 510, and the one or more programs are configured to execute the above-mentioned abnormal data processing method.

[0100] The processor 510 may include one or more processing cores. The processor 510 uses various interfaces and lines to connect various parts of the entire electronic device 500, and executes various functions of the electronic device 500 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 520, and calling data stored in the memory 520. Optionally, the processor 510 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 510 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It is understandable that the above-mentioned modem may not be integrated into the processor 510, but may be implemented separately through a communication chip.

[0101] The memory 520 may include a random access memory (RAM) or a read-only memory (ROM). The memory 520 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 520 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The data storage area may also store data created by the electronic device 500 during use, etc.

[0102] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit its protection scope. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that after reading the present invention, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the protection scope of the pending claims of the invention.

Claims

1. A method for processing abnormal data, characterized in that: The method comprises: Inputting first data to be processed into a pre-trained codec to obtain second data output by the codec; the first data and the second data both include multiple values; The target interval is determined by taking the minimum value of the second data as the lower limit of the target interval and taking the maximum value of the second data as the upper limit of the target interval; Determine the values ​​of the first data that are outside the target interval as abnormal data, and remove the abnormal data; The method further comprises: training the codec; The training of the codec includes: inputting sample data into the codec to obtain a first output result of the codec; the sample data includes a plurality of sample values, wherein the plurality of sample values ​​include a plurality of normal values ​​and one or more abnormal values; inputting the first output result into a pre-trained classification model to obtain a predicted classification result output by the classification model; determining a first loss value according to the first output result and the normal values ​​and abnormal values ​​in the sample data; determining a second loss value according to the predicted classification result and the actual classification result corresponding to the sample data; and updating the codec according to the first loss value and the second loss value; The determining a first loss value according to the first output result and normal values ​​and abnormal values ​​in the sample data includes: The first loss value is determined according to the magnitude relationship between the maximum value in the first output result and the maximum normal value in the sample data, the magnitude relationship between the maximum value in the first output result and the first abnormal value in the sample data, and / or the magnitude relationship between the minimum value in the first output result and the minimum normal value in the sample data, and the magnitude relationship between the minimum value in the first output result and the second abnormal value in the sample data; wherein the first abnormal value refers to the minimum abnormal value among the abnormal values ​​greater than the maximum normal value, and the second abnormal value refers to the maximum abnormal value among the abnormal values ​​less than the minimum normal value; The predicted classification result and the actual classification result are both in vector form; and determining the second loss value according to the predicted classification result and the actual classification result corresponding to the sample data includes: Calculating a vector distance between the predicted classification result and the actual classification result; and determining the second loss value according to the vector distance; The updating of the codec according to the first loss value and the second loss value comprises: Determine a third loss value based on a first weight corresponding to the first loss value, the first loss value, a second weight corresponding to the second loss value, and the second loss value; and update the codec using the third loss value; wherein the first weight and / or the second weight is determined based on predicted classification results and actual classification results in the most recent N trainings.

2. According to the method of claim 1, the number of the classification models is multiple, and the classification tasks performed by each classification model are different; the second loss value is determined according to the predicted classification result and the actual classification result corresponding to the sample data, comprising: The second loss value is determined according to the predicted classification result of each classification model and the actual classification result corresponding to the sample data under the corresponding classification task.

3. According to the method described in claim 2, the multiple classification models include at least two of the following: a crop pest classification model, a crop disease identification model, a crop growth identification model, and a crop growth period identification model.

4. An abnormal data processing device, characterized in that: The device comprises: A data input module, used for inputting first data to be processed into a pre-trained codec to obtain second data output by the codec; the first data and the second data both include multiple values; an interval determination module, configured to use the minimum value in the second data as the lower limit of the target interval and the maximum value in the second data as the upper limit of the target interval, thereby determining the target interval; an abnormal data clearing module, used for determining the values ​​in the first data that are outside the target interval as abnormal data, and clearing the abnormal data; The device also includes a training module, which specifically includes: A first input unit is used to input sample data into the codec to obtain a first output result of the codec; the sample data includes a plurality of sample values, and the plurality of sample values ​​include a plurality of normal values ​​and one or more abnormal values; A second input unit is used to input the first output result into a pre-trained classification model to obtain a predicted classification result output by the classification model; A first loss value determining unit, configured to determine a first loss value according to the first output result and normal values ​​and abnormal values ​​in the sample data; A second loss value determining unit, used to determine a second loss value according to the predicted classification result and the actual classification result corresponding to the sample data; An updating unit, configured to update a codec according to the first loss value and the second loss value; The first loss value determination unit is specifically used to determine the first loss value according to the size relationship between the maximum value in the first output result and the maximum normal value in the sample data, the size relationship between the maximum value in the first output result and the first abnormal value in the sample data, and / or the size relationship between the minimum value in the first output result and the minimum normal value in the sample data, and the size relationship between the minimum value in the first output result and the second abnormal value in the sample data; wherein the first abnormal value refers to the minimum abnormal value among the abnormal values ​​greater than the maximum normal value, and the second abnormal value refers to the maximum abnormal value among the abnormal values ​​less than the minimum normal value; The predicted classification result and the actual classification result are both in vector form, and the second loss value determination unit is specifically used to: calculate the vector distance between the predicted classification result and the actual classification result; and determine the second loss value according to the vector distance; The updating unit is specifically used to: determine a third loss value based on a first weight corresponding to the first loss value, the first loss value, a second weight corresponding to the second loss value, and the second loss value; and update the codec using the third loss value; wherein the first weight and / or the second weight is determined based on the predicted classification results and actual classification results in the most recent N trainings.

5. An electronic device, comprising a memory and a processor; The memory is used to store application programs; The processor is used to run the application stored in the memory to implement the abnormal data processing method according to any one of claims 1 to 3.

6. A computer-readable storage medium, wherein an application is stored in the computer-readable storage medium, and when the application is executed by a processor, it is used to implement the abnormal data processing method according to any one of claims 1 to 3.

Images