Abnormal freight rate detection method and system, electronic equipment and storage medium

Abstract

The invention relates to the technical field of data detection, provides an abnormal freight rate detection method and system, electronic equipment and a storage medium, and aims to solve the technical problem that an abnormal freight rate detection system in the prior art is difficult to comprehensively cover all possible abnormal conditions. Comprising the following steps: acquiring original freight rate data, and preprocessing the original freight rate data to obtain preprocessed freight rate data; carrying out feature coding on the preprocessed freight rate data to obtain freight rate coded data, and dividing the freight rate coded data into a training set and a threshold calculation set; constructing an auto-encoder model, and training the auto-encoder model by using the training set to obtain a trained auto-encoder model; based on a dichotomy task method, calculating a detection threshold value; and performing abnormal freight rate detection processing on the freight rate data to be detected based on the trained auto-encoder model to obtain a freight rate reconstruction error, and matching the freight rate reconstruction error with a detection threshold to obtain a detection result.

Description

Technical Field

[0001] The present invention belongs to the technical field of data detection, and in particular relates to an abnormal freight rate detection method, system, electronic equipment and storage medium. Background Art

[0002] In the civil aviation industry, freight rate management is a complex and important task. With the rapid development of e-commerce, the real-time update and release of flight prices have become more frequent and dynamic. At this time, the freight rate management system not only needs to update and release flight prices in real time, but also detect abnormal freight rates.

[0003] In the prior art, an abnormal freight rate detection system is deployed in the freight rate management process, and the published prices are detected by the abnormal freight rate detection system. Once an abnormal price is released, it can be quickly discovered and modified in time.

[0004] The prior art has the following technical problems:

[0005] 1. The existing abnormal fare detection system is located at the back end of the fare publishing process. It can only be detected after the price is released to the outside world. It cannot completely prevent consumers from seeing and purchasing these abnormally priced tickets.

[0006] 2. Existing abnormal freight rate detection systems rely on expert systems and predefined rules, which are difficult to fully cover all possible abnormal situations, especially when dealing with complex or unforeseen data. In addition, as the data environment continues to change, maintaining and updating the rule base will be very time-consuming and costly. And rule-based models have problems with low efficiency and insufficient accuracy when processing large-scale data sets. Summary of the invention

[0007] The present invention provides an abnormal freight rate detection method, system, electronic device and storage medium, aiming to solve the technical problems existing in the above-mentioned prior art that the abnormal freight rate detection system needs to release the price to the outside world before it can detect; and the abnormal freight rate detection system is difficult to fully cover all possible abnormal situations, and the maintenance and updating of the rule base is very time-consuming and costly, and there is low efficiency and insufficient accuracy when processing large-scale data sets.

[0008] The technical solution of the present invention to solve the above technical problem is as follows: a method for detecting abnormal freight rates, the method comprising:

[0009] Acquire original freight rate data, and pre-process the original freight rate data to obtain pre-processed freight rate data;

[0010] Performing feature encoding on the freight rate preprocessing data to obtain freight rate encoding data, and dividing the freight rate encoding data into a training set and a threshold calculation set;

[0011] Constructing an autoencoder model, and using the training set to train the autoencoder model to obtain a trained autoencoder model;

[0012] Based on the binary classification task method, the threshold calculation set is input into the trained autoencoder model twice to obtain the detection threshold;

[0013] Based on the trained autoencoder model, abnormal freight rate detection is performed on the freight rate data to be detected to obtain a freight rate reconstruction error, and the freight rate reconstruction error is matched with the detection threshold to obtain a detection result.

[0014] Furthermore, the above-mentioned original freight rate data includes: origin code, destination code, cabin code, voyage type, price and discount.

[0015] Furthermore, the above-mentioned preprocessing of the original freight rate data specifically includes:

[0016] Split the round trip data into two one-way data, and evenly distribute the price of the round trip data to the two one-way data.

[0017] Furthermore, the feature encoding of the freight rate preprocessing data specifically includes:

[0018] The one-hot encoding method is used to feature encode multiple airline variables in each freight rate preprocessing data, and when there is an airline variable that is empty, the leave-one-out encoding method is used to fill the empty value.

[0019] Furthermore, when the airline variable is a cabin, the label encoding method is used for encoding, specifically including:

[0020] Calculate the average freight rate of each cabin space, and sort the cabin spaces according to the size of the average freight rate to obtain the cabin space sorting order;

[0021] Based on the order of the slot sorting, label coding is performed on each slot.

[0022] Further, the above-mentioned training of the autoencoder model using the training set specifically includes:

[0023] Defining an input dimension of an encoder, and inputting the training set into the encoder according to the input dimension;

[0024] Using the multiple hidden layers of the encoder to perform dimensionality reduction processing on the input freight rate encoding data, and outputting hidden feature representation data through the output layer of the encoder;

[0025] Inputting the hidden feature representation data into a decoder, performing dimension-up processing on the hidden feature representation data using multiple hidden layers of the decoder, so that the output dimension of the decoder is equal to the input dimension of the encoder, and obtaining a reconstructed input;

[0026] Calculating the loss value between the reconstructed input and the corresponding freight rate coding data using a mean square error function as a loss function;

[0027] The Adam optimization algorithm is used to update the model parameters to optimize the loss value.

[0028] Furthermore, based on the binary classification task method, the threshold calculation set is input into the trained autoencoder model twice, and the detection threshold is obtained specifically including:

[0029] Dividing the threshold calculation set into a first threshold calculation set and a second threshold calculation set on average;

[0030] Inputting the first threshold calculation set into a trained autoencoder model to obtain a first reconstructed input;

[0031] Calculating a reconstruction error between the first threshold calculation set and the first reconstruction input to obtain a first loss array;

[0032] Calculating a mean and a standard deviation of the first loss array, and calculating a first detection threshold based on the mean and the standard deviation;

[0033] Enlarging or reducing the second threshold calculation set to obtain an abnormal data set;

[0034] Inputting the abnormal data set and the second threshold calculation set into a trained autoencoder model to obtain a second reconstruction input;

[0035] Based on the second reconstruction input, calculating a second reconstruction error to obtain a second loss array;

[0036] Match the second loss array with the first detection threshold, optimize the hyperparameters of the first detection threshold, and obtain a second detection threshold; wherein the second detection threshold is the detection threshold.

[0037] In a second aspect, in order to solve the above technical problems, the present invention also provides an abnormal freight rate detection system, comprising:

[0038] A data preprocessing module is used to obtain original freight rate data, preprocess the original freight rate data, and obtain preprocessed freight rate data;

[0039] A feature coding module, used for performing feature coding on the freight rate preprocessing data to obtain freight rate coding data, and dividing the freight rate coding data into a training set and a threshold calculation set;

[0040] A model training module is used to construct an autoencoder model and train the autoencoder model using the training set to obtain a trained autoencoder model;

[0041] A threshold calculation module, for inputting the threshold calculation set into the trained autoencoder model twice based on a binary classification task method to obtain a detection threshold;

[0042] The freight rate detection module is used to perform abnormal freight rate detection processing on the freight rate data to be detected based on the trained autoencoder model, obtain the freight rate reconstruction error, match the freight rate reconstruction error with the detection threshold, and obtain the detection result.

[0043] In a third aspect, in order to solve the above-mentioned technical problem, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the abnormal freight rate detection method of the present application is implemented.

[0044] In a fourth aspect, in order to solve the above-mentioned technical problem, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the abnormal freight rate detection method of the present application is implemented.

[0045] Compared with the prior art, the present invention has the following advantages:

[0046] 1. Since abnormal data are usually scarce and diverse, it is difficult to cover all possible situations through manual annotation. The present invention uses autoencoders as an unsupervised learning model, which can learn the intrinsic structure and pattern of data from unlabeled data without relying on a large amount of labeled data for training.

[0047] 2. By analyzing the difference between the original input and the reconstructed output, the present invention can clearly understand which features or data points are considered abnormal by the model. This reconstruction error-based approach makes it easier for users to explore the specific reasons why specific data is marked as abnormal, which helps to further investigate and verify the results.

[0048] 3. The autoencoder model of the present invention is not built based on expert predefined rules, so it can better adapt to the ever-changing data environment. It has strong robustness and can reduce the impact of noise and outliers on model performance.

[0049] 4. The autoencoder model of the present invention does not require frequent updates of complex rule bases, which reduces the cost of long-term maintenance and improves the operability and sustainability of the system.

[0050] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 A schematic diagram of a process of detecting an abnormal freight rate according to an embodiment of the present invention is shown;

[0053] Figure 2 A schematic diagram of an autoencoder model according to an embodiment of the present invention is shown;

[0054] Figure 3 A schematic diagram of training an autoencoder model according to an embodiment of the present invention is shown;

[0055] Figure 4 A schematic diagram of the structure of an abnormal freight rate detection system according to an embodiment of the present invention is shown;

[0056] Figure 5 A schematic structural diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] Figure 1 A schematic diagram of a process for detecting abnormal freight rates according to an embodiment of the present invention is shown. Figure 1 As shown, an abnormal freight rate detection method according to an embodiment of the present invention includes:

[0059] Acquire original freight rate data, and pre-process the original freight rate data to obtain pre-processed freight rate data;

[0060] Performing feature encoding on the freight rate preprocessing data to obtain freight rate encoding data, and dividing the freight rate encoding data into a training set and a threshold calculation set;

[0061] Constructing an autoencoder model, and using the training set to train the autoencoder model to obtain a trained autoencoder model;

[0062] Based on the binary classification task method, the threshold calculation set is input into the trained autoencoder model twice to obtain the detection threshold;

[0063] Based on the trained autoencoder model, abnormal freight rate detection is performed on the freight rate data to be detected to obtain a freight rate reconstruction error, and the freight rate reconstruction error is matched with the detection threshold to obtain a detection result.

[0064] Optionally, the original freight rate data includes: origin code, destination code, cabin code, voyage type, price and discount.

[0065] In this example, the original data has a total of 5 million data points. First, the original data set is reshaped. It is noted that the data volume of a certain airline in the original data set is very large, accounting for more than half of the entire data set. Therefore, 50% of the data of this airline is randomly discarded to avoid overfitting the model to the fare characteristics of this airline. At the same time, the data columns containing English characters are converted to all uppercase, and the leading and trailing spaces are removed.

[0066] Optionally, preprocessing the original freight rate data specifically includes:

[0067] Split the round trip data into two one-way data, and evenly distribute the price of the round trip data to the two one-way data.

[0068] In this embodiment, for example, if the round-trip fare from "LKG" to "HKG" is 470 yuan, it is split into one-way data from "LKG" to "HKG" with a fare of 235 yuan and one-way data from "HKG" to "LKG" with a fare of 235 yuan.

[0069] After splitting the round-trip data, the duplicate data in the original data set was discarded, and finally 900,000 pieces of original data were obtained.

[0070] Optionally, the feature encoding of the freight rate preprocessing data specifically includes:

[0071] The one-hot encoding method is used to feature encode multiple airline variables in each freight rate preprocessing data, and when there is an airline variable that is empty, the leave-one-out encoding method is used to fill the empty value.

[0072] In this embodiment, for the airline variable c, it has n possible airline categories {c1, c2, ..., cn}. For any airline c i , encode it into a vector v, the formula of vector v is as follows:

[0073] v=(v1,v2,…,v n ),have

[0074] For the origin and destination, the origin code and destination code of each original data are concatenated (for example, "LKG" and "HKG" are concatenated as "LKG-HKG") and used as a new feature column for LOO encoding.

[0075] In the original data set containing m original data samples, for the i-th original data sample, the mathematical representation of LOO encoding is:

[0076]

[0077] Among them, y i Indicates freight rate, od i Indicates the origin-destination identifier, Represents an indicator function, which is used to identify that when the origin-destination pair of a certain original data sample j is the same as that of sample i, it takes 1, otherwise it takes 0. It represents the number of samples in the entire original data set that have the same origin-destination pair as the original data sample i.

[0078] In summary, this approach allows us to generate a finely tuned encoding value for each origin-destination pair of the original data sample and avoids the data leakage problem.

[0079] Optionally, if there is only one sample of the origin-destination pair of a certain category, it will cause null value problems during model training. The null value filling method through the leave-one-out encoding method includes:

[0080] If there are multiple samples with the same origin (even if the destination is different), calculate the average of the freight rates of all other samples except the current sample (i.e., the leave-one-out mean). Use this leave-one-out mean to fill in the empty values ​​in a single origin-destination pair.

[0081] If the above conditions are not met (i.e. there are not enough samples with the same origin), but there are multiple samples with the same destination, the leave-one-out mean of the freight rates for the same destination will be used to fill the gap.

[0082] If neither enough samples share the same origin nor enough samples share the same destination, it is considered that the data point cannot be filled by reasonable methods, so it is chosen to be removed.

[0083] Optionally, when the airline variable is a cabin class, a label encoding method is used for encoding, specifically including:

[0084] Calculate the average freight rate of each cabin space, and sort the cabin spaces according to the size of the average freight rate to obtain the cabin space sorting order;

[0085] Based on the order of the slot sorting, label coding is performed on each slot.

[0086] In this embodiment, the label encoding formula is as follows:

[0087]

[0088] Among them, b represents a certain cabin code. It represents the average freight rate of all samples in this space. Indicates based on The rank (or order) of b after sorting all the slots by the values ​​of , which is then used for label encoding of b. Label(b) is the unique integer label obtained after b is encoded by this special label encoding method.

[0089] In this embodiment, 90% of the freight rate code data is used as the training set, denoted as 10% of the freight rate code data is used as the threshold calculation set, denoted as We randomly select 5,000 pieces of original data from the 5 million pieces of original data as the evaluation set, denoted as

[0090] Optionally, using the training set to train the autoencoder model specifically includes:

[0091] Defining an input dimension of an encoder, and inputting the training set into the encoder according to the input dimension;

[0092] Using the multiple hidden layers of the encoder to perform dimensionality reduction processing on the input freight rate encoding data, and outputting hidden feature representation data through the output layer of the encoder;

[0093] Inputting the hidden feature representation data into a decoder, performing dimension-up processing on the hidden feature representation data using multiple hidden layers of the decoder, so that the output dimension of the decoder is equal to the input dimension of the encoder, and obtaining a reconstructed input;

[0094] Calculating the loss value between the reconstructed input and the corresponding freight rate coding data using a mean square error function as a loss function;

[0095] The Adam optimization algorithm is used to update the model parameters to optimize the loss value.

[0096] In this embodiment, Figure 2 As shown in the figure, the constructed autoencoder model mainly consists of three parts: encoder, hidden feature representation and decoder. The encoder is used to compress the input data from a high-dimensional space to a low-dimensional hidden feature space. It outputs a 3-dimensional hidden feature vector, which attempts to capture the key features of the original input data. The decoder is used to restore the hidden feature vector to the dimension of the original data.

[0097] Among them, the encoding process of the encoder is shown as follows:

[0098] h i =f(x i )

[0099] Among them, h i ∈R q , represents the q-dimensional hidden layer vector, f:R n →R q , represents the encoder.

[0100] The decoding process of the decoder is shown as follows:

[0101]

[0102] in, Represents the output of the autoencoder model.

[0103] In summary, the training process of the autoencoder model is to find the encoder f and the decoder g so that after the input data x passes through the encoder and the decoder, the reconstructed data As close as possible to the original input data x. Mathematically, we want to minimize the expected value of the input-output loss. The expected value is shown below:

[0104]

[0105] Among them, Δ represents the difference between input and output obtained by the loss function. <·> represents the average value. Here it refers to the average value of all loss values. After the model is trained, there should be

[0106] like Figure 3 As shown in the figure, the input layer of the encoder receives a 40-dimensional vector, which is reduced to 32 dimensions through the first hidden layer, further reduced to 16 dimensions through the second hidden layer, and reduced to 3-dimensional hidden representation through the third hidden layer to obtain hidden feature representation data; in the dimensionality reduction process, each hidden layer uses the RELU activation function to realize nonlinear transformation between layers and only retain positive outputs.

[0107] The decoder gradually increases the dimension of the 3-dimensional hidden representation from 16 dimensions through the fourth hidden layer, increases the dimension from 16 dimensions to 32 dimensions through the fifth hidden layer, and restores the 32 dimensions to the original 40 dimensions through the sixth hidden layer. In the process of increasing the dimension, for the first n-dimensional outputs, since they represent the unique hot encoding of the airlines, the sigmoid function is used to constrain their values ​​between 0 and 1. For the remaining 40-n dimensions, considering that their business meanings need to remain positive, the RELU function is continued to be used to ensure the positivity of the output.

[0108] The training steps are:

[0109] Initialize the encoder and decoder weights;

[0110] Input the freight rate encoded data into the autoencoder model to obtain the reconstructed input;

[0111] Use mean square error (MSE) as the loss function to calculate the input and reconstructed input capacitance loss values;

[0112] The gradient is calculated according to the loss function and the parameters of the encoder and decoder are updated through the Adam optimizer.

[0113] Repeat the above steps until a complete traversal of all training data is completed.

[0114] During the training process, the learning rate was set to 0.001, the mean square error function was used as the loss function, the Adam optimizer was used, each batch of training data was 32, and the number of training rounds was set to 10 rounds.

[0115] Optionally, based on the binary classification task method, the threshold calculation set is input into the trained autoencoder model twice to obtain the detection threshold, which specifically includes:

[0116] Dividing the threshold calculation set into a first threshold calculation set and a second threshold calculation set on average;

[0117] Inputting the first threshold calculation set into a trained autoencoder model to obtain a first reconstructed input;

[0118] Calculating a reconstruction error between the first threshold calculation set and the first reconstruction input to obtain a first loss array;

[0119] Calculating a mean and a standard deviation of the first loss array, and calculating a first detection threshold based on the mean and the standard deviation;

[0120] Enlarging or reducing the second threshold calculation set to obtain an abnormal data set;

[0121] Inputting the abnormal data set and the second threshold calculation set into a trained autoencoder model to obtain a second reconstruction input;

[0122] Based on the second reconstruction input, calculating a second reconstruction error to obtain a second loss array;

[0123] Match the second loss array with the first detection threshold, optimize the hyperparameters of the first detection threshold, and obtain a second detection threshold; wherein the second detection threshold is the detection threshold.

[0124] In this embodiment, the mean square error function is selected to calculate the reconstruction error, and the formula is shown as follows:

[0125]

[0126] The formula for calculating the first detection threshold is as follows:

[0127] θ=μ+λ·σ

[0128] Among them, μ represents the mean, λ represents the parameter, and σ represents the standard deviation;

[0129] Threshold calculation set is evenly divided into the first threshold calculation set and the second threshold calculation set In order to determine the appropriate reconstruction error threshold, the hyperparameter λ is adjusted by artificially creating abnormal data, that is, the second threshold calculation set is randomly Zoom in to 10 to 20 times the original size, or zoom out to 0 to 0.2 times the original size to get an abnormal data set Then the second threshold calculation set and abnormal data sets Input the autoencoder model separately and calculate the reconstruction error of each data. According to the current threshold θ, judge whether each data is abnormal: if the reconstruction error is greater than the threshold θ, it is inferred that it is abnormal data, otherwise it is inferred that it is normal data. Evaluate the accuracy of the binary classification task under different λ values, select the λ value that maximizes the accuracy of the binary classification task, and obtain the detection threshold.

[0130] When inferring whether a new freight rate is an abnormal freight rate on a new platform, first build a new model with the same architecture based on the trained model weights, then perform feature encoding on the freight rate data, and finally input the feature-encoded data into the model to calculate the reconstruction error. If the reconstruction error is greater than the detection threshold, it is considered abnormal data, otherwise it is considered normal data.

[0131] Based on Figure 1 Based on the same principle as the method shown in , the embodiment of the present invention also provides an abnormal freight rate detection system, such as Figure 4 As shown in, including:

[0132] A data preprocessing module is used to obtain original freight rate data, preprocess the original freight rate data, and obtain preprocessed freight rate data;

[0133] A feature coding module, used for performing feature coding on the freight rate preprocessing data to obtain freight rate coding data, and dividing the freight rate coding data into a training set and a threshold calculation set;

[0134] A model training module is used to construct an autoencoder model and train the autoencoder model using the training set to obtain a trained autoencoder model;

[0135] A threshold calculation module, for inputting the threshold calculation set into the trained autoencoder model twice based on a binary classification task method to obtain a detection threshold;

[0136] The freight rate detection module is used to perform abnormal freight rate detection processing on the freight rate data to be detected based on the trained autoencoder model, obtain the freight rate reconstruction error, match the freight rate reconstruction error with the detection threshold, and obtain the detection result.

[0137] The abnormal freight rate detection system of the embodiment of the present invention can execute the abnormal freight rate detection method provided by the embodiment of the present invention, and the implementation principle thereof is similar. The actions executed by each module and unit in the abnormal freight rate detection system in each embodiment of the present invention correspond to the steps in the abnormal freight rate detection method in each embodiment of the present invention. For the detailed functional description of each module of the abnormal freight rate detection system, please refer to the description in the corresponding abnormal freight rate detection method shown in the foregoing text, which will not be repeated here.

[0138] The abnormal freight rate detection system may be a computer program (including program code) running in a computer device. For example, the abnormal freight rate detection system is an application software. The application software may be used to execute the corresponding steps in the method provided in the embodiment of the present invention.

[0139] The modules involved in the embodiments of the present invention may be implemented by software or hardware, wherein the name of a module does not limit the module itself in some cases.

[0140] Based on the same principle as the method shown in the embodiments of the present invention, an electronic device is also provided in the embodiments of the present invention, which may include but is not limited to: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the method shown in any embodiment of the present invention by calling the computer program.

[0141] In an alternative embodiment, an electronic device is provided, such as Figure 3 As shown, Figure 3The electronic device shown includes: a processor and a memory. The processor and the memory are connected, such as through a bus. Optionally, the electronic device may also include a transceiver, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver is not limited to one, and the structure of the electronic device does not constitute a limitation on the embodiments of the present invention.

[0142] The memory is used to store the application code (computer program) for executing the solution of the present invention, and the execution is controlled by the processor. The processor is used to execute the application code stored in the memory to implement the content shown in the above method embodiment.

[0143] The electronic device may also be a terminal device. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0144] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding contents of the aforementioned method embodiment.

[0145] According to another aspect of the present invention, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the implementation modes of the above-mentioned various embodiments.

[0146] It should be understood that the flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0147] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0148] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure scope involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) by each other.

Claims

1. A method for detecting abnormal freight rates, characterized in that: The method comprises: Acquire original freight rate data, and pre-process the original freight rate data to obtain pre-processed freight rate data; Performing feature encoding on the freight rate preprocessing data to obtain freight rate encoding data, and dividing the freight rate encoding data into a training set and a threshold calculation set; Constructing an autoencoder model, and using the training set to train the autoencoder model to obtain a trained autoencoder model; Based on the binary classification task method, the threshold calculation set is input into the trained autoencoder model twice to obtain the detection threshold; Based on the trained autoencoder model, abnormal freight rate detection is performed on the freight rate data to be detected to obtain a freight rate reconstruction error, and the freight rate reconstruction error is matched with the detection threshold to obtain a detection result.

2. The abnormal freight rate detection method according to claim 1, characterized in that: The original freight rate data includes: origin code, destination code, cabin code, voyage type, price and discount.

3. The abnormal freight rate detection method according to claim 1, characterized in that: Preprocessing the original freight rate data specifically includes: Split the round trip data into two one-way data, and evenly distribute the price of the round trip data to the two one-way data.

4. The abnormal freight rate detection method according to claim 1, characterized in that: The feature encoding of the freight rate preprocessing data specifically includes: The one-hot encoding method is used to feature encode multiple airline variables in each freight rate preprocessing data, and when there is an airline variable that is empty, the leave-one-out encoding method is used to fill the empty value.

5. The abnormal freight rate detection method according to claim 4, characterized in that: When the airline variable is a cabin class, the label encoding method is used for encoding, including: Calculate the average freight rate of each cabin space, and sort the cabin spaces according to the size of the average freight rate to obtain the cabin space sorting order; Based on the order of the slot sorting, label coding is performed on each slot.

6. The abnormal freight rate detection method according to claim 1, characterized in that: Using the training set to train the autoencoder model specifically includes: Defining an input dimension of an encoder, and inputting the training set into the encoder according to the input dimension; Using the multiple hidden layers of the encoder to perform dimensionality reduction processing on the input freight rate encoding data, and outputting hidden feature representation data through the output layer of the encoder; Inputting the hidden feature representation data into a decoder, performing dimension-up processing on the hidden feature representation data using multiple hidden layers of the decoder, so that the output dimension of the decoder is equal to the input dimension of the encoder, and obtaining a reconstructed input; Calculating the loss value between the reconstructed input and the corresponding freight rate coding data using a mean square error function as a loss function; The Adam optimization algorithm is used to update the model parameters to optimize the loss value.

7. The abnormal freight rate detection method according to claim 1, characterized in that: Based on the binary classification task method, the threshold calculation set is input into the trained autoencoder model twice, and the detection threshold is obtained specifically including: Dividing the threshold calculation set into a first threshold calculation set and a second threshold calculation set on average; Inputting the first threshold calculation set into a trained autoencoder model to obtain a first reconstructed input; Calculating a reconstruction error between the first threshold calculation set and the first reconstruction input to obtain a first loss array; Calculating a mean and a standard deviation of the first loss array, and calculating a first detection threshold based on the mean and the standard deviation; Enlarging or reducing the second threshold calculation set to obtain an abnormal data set; Inputting the abnormal data set and the second threshold calculation set into a trained autoencoder model to obtain a second reconstruction input; Based on the second reconstruction input, calculating a second reconstruction error to obtain a second loss array; Match the second loss array with the first detection threshold, optimize the hyperparameters of the first detection threshold, and obtain a second detection threshold; wherein the second detection threshold is the detection threshold.

8. An abnormal freight rate detection system, characterized in that: include: A data preprocessing module is used to obtain original freight rate data, preprocess the original freight rate data, and obtain preprocessed freight rate data; A feature coding module, used for performing feature coding on the freight rate preprocessing data to obtain freight rate coding data, and dividing the freight rate coding data into a training set and a threshold calculation set; A model training module is used to construct an autoencoder model and train the autoencoder model using the training set to obtain a trained autoencoder model; A threshold calculation module, for inputting the threshold calculation set into the trained autoencoder model twice based on a binary classification task method to obtain a detection threshold; The freight rate detection module is used to perform abnormal freight rate detection processing on the freight rate data to be detected based on the trained autoencoder model, obtain the freight rate reconstruction error, match the freight rate reconstruction error with the detection threshold, and obtain the detection result.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images