Asset matching detection method and device

Abstract

The invention discloses an asset matching detection method and device. The method comprises the steps that an asset false matching detection model is initialized, and the weight of the asset false matching detection model is loaded; under the condition that the asset data input into the asset false matching detection model are two pictures, respectively carrying out normalization processing on the two pictures, carrying out similarity prediction on the two pictures by using the asset false matching detection model, and outputting a predicted similarity score; under the condition that the asset data input into the asset false matching detection model are two videos, respectively carrying out normalization processing on the two videos, selecting a frame pair from the two videos subjected to the normalization processing to obtain two pictures corresponding to the frame pair, performing similarity prediction on the two pictures by using the asset false matching detection model, and outputting a predicted similarity score; and judging whether the asset data is matched or not based on the similarity score. The technical problem of inaccurate asset matching is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an asset matching detection method and device. Background Art

[0002] As asset management and supervision continue to strengthen, the requirements for data security, asset security, and information security are increasing. However, due to remote management and the existence of diversified investment entities, such as power plants and large railway equipment, the matching and identification of fixed assets faces a series of challenges such as low efficiency, high labor costs, and false matching. In response to these problems, an intelligent and automated method is urgently needed to identify, match, and monitor fixed assets to meet the evolving business needs.

[0003] Image similarity detection technology can help with asset matching and auditing. Image similarity detection refers to determining the similarity between two images. In this process, it is necessary to extract the features of the two images and compare these features to calculate the similarity between the two images. This is usually achieved by calculating the feature vectors or feature descriptors of the two images. In the study of similarity calculation, image similarity detection can be divided into traditional learning methods and deep learning methods.

[0004] Traditional learning methods usually include two steps: feature extraction and similarity measurement. Feature extraction can be divided into similarity detection methods based on local features and global features. Among them, the more representative algorithms are based on the scale invariant feature transform feature (SIFT) algorithm, the accelerated robust features (SURF) algorithm and the oriented fast and rotated brief (ORB) algorithm. Similarity detection methods based on global features usually use color histograms, gray level co-occurrence matrices, etc. to directly describe the global features of the image. Similarity measurement is another important research direction. The goal of similarity measurement is to map two feature vectors to a real number through a function to judge the similarity between two images. Commonly used similarity measurement methods include Euclidean distance, cosine similarity, Hamming distance, etc. These methods achieve recognition by extracting features from the shallow surface of the image, but their feature extraction capabilities are weak. They are only effective in simple environments for recognition verification and are unable to extract deep image features. In other complex environments such as changes in lighting and angles, recognition precision and accuracy will decrease.

[0005] In recent years, the application of deep learning technology in image similarity detection has made significant progress and has become the mainstream method in the field of image similarity detection. Deep learning methods use neural networks to achieve end-to-end learning of feature representation and similarity measurement. For example, Siamese Neural Networks (SNN) have performed well in many image similarity detection tasks and have gradually been widely used in various image similarity comparison and metric learning tasks. However, the existing image similarity measurement methods based on deep learning still have the technical problem of inaccurate measurement.

[0006] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0007] The embodiments of the present invention provide an asset matching detection method and device to at least solve the technical problem of inaccurate asset matching.

[0008] According to one aspect of an embodiment of the present invention, there is provided an asset matching detection method, comprising: initializing an asset false matching detection model, loading the weight of the asset false matching detection model; when the asset data input to the asset false matching detection model is two pictures, normalizing the two pictures respectively, using the asset false matching detection model to perform similarity prediction on the two pictures, and outputting predicted similarity scores; when the asset data input to the asset false matching detection model is two videos, normalizing the two videos respectively, selecting frame pairs from the two normalized videos to obtain two pictures corresponding to the frame pairs, using the asset false matching detection model to perform similarity prediction on the two pictures, and outputting predicted similarity scores; and judging whether the asset data matches based on the similarity scores.

[0009] According to another aspect of an embodiment of the present invention, there is also provided an asset matching detection device, comprising: an initialization module, configured to initialize an asset false matching detection model, and load the weight of the asset false matching detection model; a detection module, configured to: when the asset data input to the asset false matching detection model is two pictures, respectively normalize the two pictures, and use the asset false matching detection model to perform similarity prediction on the two pictures, and output predicted similarity scores; when the asset data input to the asset false matching detection model is two videos, respectively normalize the two videos, select frame pairs from the two normalized videos to obtain two pictures corresponding to the frame pairs, and use the asset false matching detection model to perform similarity prediction on the two pictures, and output predicted similarity scores; a determination module, configured to determine whether the asset data matches based on the similarity scores.

[0010] In an embodiment of the present invention, an asset false match detection model is initialized, and the weight of the asset false match detection model is loaded; when the asset data input to the asset false match detection model is two pictures, the two pictures are normalized respectively, and the asset false match detection model is used to predict the similarity of the two pictures, and the predicted similarity score is output; when the asset data input to the asset false match detection model is two videos, the two videos are normalized respectively, and frame pairs are selected from the two normalized videos to obtain two pictures corresponding to the frame pairs, and the asset false match detection model is used to predict the similarity of the two pictures, and the predicted similarity score is output; based on the similarity score, it is judged whether the asset data matches. Through the above scheme, the technical problem of inaccurate asset matching is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0012] Figure 1 is a flow chart of an asset matching detection method according to an embodiment of the present invention;

[0013] Figure 2 is a flow chart of a method for training an asset false match detection model according to an embodiment of the present invention;

[0014] Figure 3 is a flow chart of a method for verifying an asset false match detection model according to an embodiment of the present invention;

[0015] Figure 4 is a flow chart of another method for training an asset false matching detection model based on deep learning according to an embodiment of the present invention;

[0016] Figure 5 is an architectural diagram of an asset false match detection model according to an embodiment of the present invention;

[0017] Figure 6 is a structural diagram of a twin neural network according to an embodiment of the present invention;

[0018] Figure 7 is a residual network structure diagram according to an embodiment of the present invention;

[0019] Figure 8 is a structural diagram of a CBAM attention mechanism according to an embodiment of the present invention;

[0020] Fig. 9 is a diagram of the ResNet50 network structure according to an embodiment of the present invention that incorporates an attention mechanism;

[0021] Fig.10 is a model structure diagram of a fused convolutional block attention module and a twin network according to an embodiment of the present invention;

[0022] Fig.11 is a flow chart of another asset matching detection method according to an embodiment of the present invention;

[0023] Fig.12 is a flow chart of another verification method of an asset false match detection model according to an embodiment of the present invention;

[0024] Fig.13 is a sample graph of a self-constructed asset dataset according to an embodiment of the present invention;

[0025] Fig.14 is an example diagram of data enhancement according to an embodiment of the present invention;

[0026] Fig.15 is a curve diagram showing changes in training loss and accuracy versus the number of iterations according to an embodiment of the present invention;

[0027] Fig.16 It is a structural schematic diagram of a training device for an asset false matching detection model based on deep learning according to an embodiment of the present invention;

[0028] Fig.17 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] According to an embodiment of the present invention, a method embodiment of a method for training an asset false match detection model based on deep learning is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0032] Overview

[0033] In order to solve the problems of low efficiency, high labor cost and false matching of overseas asset matching under remote management and diversified investment entities, a false matching detection method based on Siamese neural network and convolutional attention module is proposed. The Siamese neural network is used as the basic framework to build a convolutional neural network model and introduce the CBAM (Convolutional Block Attention Module) hybrid attention mechanism to form a feature extraction module, and the pre-trained weights are transferred to the training model through the transfer learning method. At the same time, in order to make the model have better generalization ability, data enhancement is used to process the samples. Experiments on asset matching tasks with different types of asset data collected show that the proposed model has achieved an accuracy of 92.42% and an F1 measure value of 92.48%, which can improve the recognition accuracy of false asset matching and bring more efficient and intelligent solutions to the field of asset management.

[0034] Example 1

[0035] This application provides an asset matching detection method, such as Figure 1 As shown, the method comprises the following steps:

[0036] Step S102, initializing an asset false match detection model and loading the weight of the asset false match detection model;

[0037] The asset false matching detection model is obtained by using the twin neural network as the basic framework to build a convolutional neural network and introduce a hybrid attention mechanism. The specific training method is as follows: Figure 2 As shown, the following steps are included:

[0038] Step S1022, fusing the convolutional neural network combined with the attention mechanism and the twin network to construct the asset false matching detection model;

[0039] The convolutional neural network is constructed using the twin neural network as a basic framework; the attention mechanism is introduced into the convolutional neural network to construct the asset false matching detection model. The pre-trained weights are transferred to the asset false matching detection model through a transfer learning method.

[0040] Step S1024, inputting the training data into the convolutional neural network, and using the convolutional neural network to extract features from two sample images that form positive samples or negative samples in the training data to obtain two feature vectors;

[0041] First, asset data is collected and divided into a training data set, a validation data set, and a test data set; the training data set is preprocessed, wherein the preprocessing includes at least one of the following: image normalization, size adjustment, and data enhancement.

[0042] The convolutional neural network is used to perform feature extraction on the two sample images respectively to obtain two high-dimensional feature maps as the two feature vectors, wherein the high-dimensional feature map is a feature map with a dimension greater than a preset dimension.

[0043] Step S1026, comparing the feature difference between the two feature vectors, and predicting the similarity between the two sample images based on the feature difference;

[0044] The L1 distance between the two feature vectors is calculated using a mapping function as the feature difference; the L1 distance is compared and fused using a fully connected layer to measure the similarity between the two sample images; and based on the similarity, the predicted similarity is output using an activation function.

[0045] Step S1028, using the loss function to compare the predicted similarity and the actual similarity, obtain the loss between the predicted similarity and the actual similarity, and update the network parameters of the convolutional neural network and the twin network based on the loss to obtain the trained asset false match detection model.

[0046] Step S104, when the asset data input to the asset false match detection model is two pictures, normalizing the two pictures respectively, and using the asset false match detection model to perform similarity prediction on the two pictures, and outputting the predicted similarity score;

[0047] First, the convolutional neural network is used to extract features from the two images; for example, multiple filters are used to scan the two images respectively to generate feature maps, and the size of the feature maps is reduced by downsampling; an activation function is used to introduce nonlinear features into the feature maps with reduced sizes, and the feature maps with introduced nonlinear features are used as the extracted features.

[0048] Next, the weight of the feature map is adjusted. The channel attention module is used to capture the importance of each channel through global average pooling and global maximum pooling, and the channel weight is generated through a multi-layer perceptron; the spatial attention module is used to pool the feature map in the channel dimension and generate spatial weights through convolution operations; the weight of the feature map is dynamically adjusted based on the channel weight and the spatial weight.

[0049] Finally, the metric function of the twin neural network is used to perform similarity measurement on the extracted features, and the predicted similarity score is output. For example, the metric function is used to calculate the distance between two feature vectors corresponding to the extracted features respectively; based on the distance, the similarity measurement is performed and the predicted similarity score is output.

[0050] Step S106, when the asset data input to the asset false match detection model is two videos, normalize the two videos respectively, select frame pairs from the two normalized videos to obtain two pictures corresponding to the frame pairs, and use the asset false match detection model to predict the similarity of the two pictures, and output the predicted similarity scores;

[0051] First, two pictures are extracted from the video. The two videos are normalized respectively. The purpose of normalization is to eliminate the interference factors introduced in the video due to differences in shooting conditions, equipment or environment, so that the video data is consistent in the same feature space, thereby improving the detection accuracy of the model. Specifically, the normalization process can include operations such as resizing, color correction, brightness and contrast adjustment of the video frame to ensure that the two videos are compared under the same benchmark. After the normalization process is completed, a number of frame pairs are selected from the two normalized videos according to a certain strategy. The frame pair selection strategy can be determined based on the time interval, key frame extraction algorithm or other feature matching algorithm to ensure that the selected frame pairs can represent the main content and features of the video. The selected frame pairs usually contain two frames corresponding in time, and each frame pair corresponds to one frame in the two videos. Finally, the selected frame pairs are converted into picture form for further comparison and analysis. By extracting the video frames as static pictures, it is more convenient to use image processing algorithms to analyze them, so as to determine whether there is a false match between the two videos. This process ensures that when the asset false match detection model is fed into it, it processes uniform and representative image data, thereby improving the accuracy and reliability of detection.

[0052] Next, similarity prediction is performed. The similarity prediction method is the same as step S104 and will not be described again here.

[0053] Step S108, judging whether the asset data matches based on the similarity score.

[0054] In some embodiments, after the asset false match detection model is trained, the asset false match detection model can also be verified. Figure 3 As shown, the verification method of the asset false matching detection model includes the following steps:

[0055] Step S302, acquiring asset data, and dividing the asset data into a training data set, a verification data set, and a test data set;

[0056] Step S304, based on the test data set, a quantitative analysis method is used to evaluate the detection performance of the asset false match detection model on the asset image similarity using accuracy, precision, recall, and F1 measurement value to verify the asset false match detection model.

[0057] The training data set is preprocessed, wherein the preprocessing includes data enhancement using the number of operations and the magnitude of the transformation. The data enhancement includes at least one of the following: brightness adjustment, contrast adjustment, rotation, scaling, flipping, and occlusion.

[0058] In some embodiments, based on a binary confusion matrix, the predicted positive class TP and the predicted negative class FN are determined when the test data in the test data set is a true positive class, as well as the predicted positive class FP and the predicted negative class TN are determined when the test data in the test data set is a true negative class; based on at least part of the predicted positive class TP and the predicted negative class FN in the case of the true positive class, and the predicted positive class FP and the predicted negative class TN in the case of the true negative class, the accuracy, the precision, and the recall rate are determined, and the F1 measurement value is determined based on the accuracy and the recall rate; based on the accuracy, the precision, the recall rate, and the F1 measurement value, the detection performance of the asset false match detection model on the asset image similarity is evaluated.

[0059] In some embodiments, while controlling the variables, the backbone network, the improved method and the data enhancement of the asset false match detection model are adjusted respectively; for the adjusted backbone network, the improved method and the data enhancement, the accuracy rate, the precision rate, the recall rate, and the F1 measurement value are calculated respectively to evaluate the impact of the backbone network, the improved method and the data enhancement on the performance of the asset false match detection model.

[0060] In some embodiments, under the same test data set, a color histogram method is used to extract color features of images in the test data set, and a gray-level co-occurrence matrix method is used to extract texture features of images, and the similarity of images is calculated based on the extracted color features and texture features, respectively. Based on the calculated similarities, the accuracy, precision, recall and F1 measurement values ​​of the color histogram method and the gray-level co-occurrence matrix method are respectively calculated; the accuracy, precision, recall and F1 measurement values ​​of the color histogram method and the gray-level co-occurrence matrix method are compared with the accuracy, precision, recall and F1 measurement values ​​of the asset false match detection model to verify the asset false match detection model.

[0061] Example 2

[0062] Figure 4 is another method for training an asset false matching detection model based on deep learning according to an embodiment of the present invention, such as Figure 4 As shown, the method includes:

[0063] Step S402: data preprocessing.

[0064] First, we collect asset data and divide the dataset into training set, validation set and test set, and preprocess part of the dataset. The preprocessing steps include image normalization, resizing, data augmentation, etc. to improve the generalization ability and robustness of the model.

[0065] Step S404: model training.

[0066] The asset false matching detection model provided by this application is as follows Figure 5 The result is a fusion of a convolutional neural network with an attention mechanism and a twin network.

[0067] The Siamese network performs matching evaluation by calculating the similarity of the input asset images. Considering that the input is two images, this application selects the Siamese Neural Network as the network framework for asset similarity matching. Figure 6 , Siamese Neural Network is a neural network architecture used to compare the similarity of two input samples. It consists of two sub-networks with shared weights. Each sub-network receives an input sample and maps it to the feature space. The similarity of the samples is determined by calculating the distance between the two feature vectors. The input images are X1, X2, GW(X1), GW(X2) are the extracted feature vectors, and the mapping function EW is calculated by the modulus length of the difference between the two vectors. The closer its value is to 1, the higher the similarity.

[0068] Compared with other network models, the twin network only needs to learn a universal embedding function from a dataset containing enough samples in the initial stage, and can directly use it for the recognition of test data without retraining. Due to its unique structure, an effective model can be trained even with small dataset images. The subnetwork of the twin network is usually a convolutional neural network, which makes it easy to select a suitable network structure according to different image recognition tasks, and is suitable for matching and identifying assets.

[0069] Convolutional neural networks are used for feature extraction. Feature extraction and description are key issues in computer vision. They mainly involve extracting useful and distinguishable features from images. These features can be used to achieve target detection, image recognition and other related tasks. In solving the problem of asset false match detection, feature extraction and description are crucial links. The core of asset false match detection is to identify false matches by analyzing and comparing the features of different assets. Convolutional Neural Network (CNN) is a powerful tool for image feature extraction.

[0070] Convolutional neural networks consist of five hierarchical structures: input layer, convolution layer, pooling layer, activation layer, and fully connected layer. The input layer receives the original image data; the convolution layer uses multiple filters to scan the image and generate feature maps; the pooling layer reduces the size of the feature map through downsampling operations, thereby reducing the amount of calculation and parameters; the activation layer usually uses activation functions such as ReLU (Rectified Linear Unit) to introduce nonlinear features; the fully connected layer accepts the feature input from the previous layer of the network and maps it to an output vector with another fixed dimension. As the number of layers in the network increases, convolutional neural networks are able to extract more advanced and abstract features. However, when the depth of the network increases to a certain extent, the gradient disappearance or gradient explosion problem will occur. This makes it difficult for deep networks to converge during training, affecting the performance of the model. To address this problem, the ResNet (residual network) model came into being. The core idea of ​​ResNet is to introduce residual blocks and solve the degradation problem in deep neural network training through residual connections (skip connections). Figure 7 The structure of the residual block, a basic unit in the ResNet model, is shown.

[0071] Among them, x is the input of the convolutional neural network, and the output H(x) is obtained after a series of weight layers and nonlinear activation functions. The traditional forward propagation process can be expressed as: H(x) = F(x). The key to the residual block is the introduction of a skip connection or residual connection. This connection directly transmits the input x to the output end and adds it to the output F(x) processed by the convolution layer and the activation function, that is, H(x) = F(x) + x. This form of design makes the module learn the residual F(x)-x instead of the entire function F(x), so it is called a residual network.

[0072] This application uses the ResNet50 model structure, which can extract multi-layer feature information and is suitable for asset false matching models. Its network structure is shown in Table 1.

[0073] Table 1 ResNet50 network structure

[0074]

[0075] In order to deal with influencing factors such as irrelevant background noise in asset images, this application introduces the CBAM module in the ResNet50 model, which can effectively enhance the feature extraction capability and improve the accuracy of asset false match detection.

[0076] CBAM (Convolutional Block Attention Module) is an attention mechanism module used to enhance the performance of convolutional neural networks. It introduces two sub-modules, Channel Attention and Spatial Attention, to weight feature maps in the channel and spatial dimensions respectively. The channel attention module captures the importance of each channel through global average pooling and global maximum pooling, and generates channel weights through a multi-layer perceptron (MLP). The spatial attention module pools the feature map in the channel dimension and generates spatial weights through convolution operations to highlight important spatial areas. Through these two steps, CBAM dynamically adjusts the weights of the feature map, allowing the network to focus on key features more accurately, improving the representation ability and performance of the model, and is widely used in tasks such as image classification, object detection, and similarity matching.

[0077] CBAM structure diagram Figure 8 As shown, the input feature map F, the overall calculation process is shown in formula (1) and formula (2):

[0078]

[0079] in, represents the dot product operation; F' represents the output of the input feature map F through the channel attention module; F" is the output of F' through the spatial attention module. The operation formulas of the channel attention module and the spatial attention module are shown in Equation (3) and Equation (4), where σ represents the Sigmoid function:

[0080] M c (F) = σ(MLP(AvgPool(F))+MLP(MaxPool(F))) (3)

[0081] M s (F′) = σ(Conv([AvgPool(F′);MaxPool(F′)])) (4)

[0082] Transfer learning helps train reliable decision functions in the target domain by transferring auxiliary source knowledge, thereby solving the learning problem when the sample data in the target domain is unlabeled or has only a small number of labeled samples. In the training process of this application, the pre-trained model of the ResNet50 network is called by transfer learning. ResNet50 is a deep convolutional neural network model pre-trained on the ImageNet dataset with rich feature representation capabilities. By fine-tuning its weight parameters on the source training set, asset feature vectors can be effectively extracted, thereby reducing resource consumption and improving training efficiency. The feature extraction part of this application combines ResNet50 and CBAM, Fig. 9 The ResNet50 network structure with integrated attention mechanism is shown.

[0083] The ResNet50 network combined with the attention mechanism and the twin network are integrated. The structural diagram of the network model is as follows Fig.10 As shown in the figure. In this dual-branch ResNet50_CBAM network, an asset image is input to form a positive sample or a negative sample, and then feature extraction is performed to map the sample to a low-dimensional space. Then, the extracted feature vector is flattened into a one-dimensional vector, the L1 distance between the two vectors is calculated, and two fully connected layers are used to compare and fuse features to better measure similarity. Finally, the Sigmoid function is used to output the similarity to determine whether the asset matches.

[0084] The present application reversely updates the asset false match detection network through a loss function, and adjusts the parameters of the feature extraction layer and the fully connected layer so that it can more accurately predict the similarity of the input asset image pair. The fully connected neural network outputs a similarity score for the image pair between 0 and 1, and then performs a cross entropy operation on the score and the label y of the input image pair (where y=1 indicates similarity and y=0 indicates dissimilarity). The cross entropy loss function measures the difference between the model output and the true label. The present application determines the binary cross entropy loss of the current training data based on the actual similarity and predicted similarity of the current training data in the current training data set; accumulates all binary cross entropy losses in the training data set and averages all accumulated binary cross entropy losses to obtain the final loss. The specific calculation process is as follows:

[0085] First, the output of the model is compressed to [0, 1] by the Sigmoid activation function, which represents the probability of predicting the positive class. The Sigmoid function is shown in formula (5):

[0086]

[0087] Then, the predicted probability is compared with the true label and the binary cross entropy loss is calculated as shown in formula (6):

[0088] BCE = -[y i ·log(σ(z i ))+(1-y i )·log(1-σ(z i ))] (6)

[0089] Finally, the losses of all samples are averaged to obtain the final loss value, as shown in formula (7):

[0090]

[0091] Where N is the number of image pairs, yi is the true label of the i-th image pair, i.e., the actual similarity (0 or 1), z i is the predicted similarity of the model.

[0092] The scheme uses the loss function to reversely update the asset false match detection network and adjust the parameters of the feature extraction layer and the fully connected layer to improve the detection accuracy. Specifically, the output of the model is compressed to [0,1] by the Sigmoid activation function, indicating the probability of predicting the positive class. Then the output similarity score is cross-entropy calculated with the label of the input image pair (y=1 for similarity and y=0 for dissimilarity). The cross-entropy loss function measures the difference between the model output and the true label, optimizes the network parameters through back propagation, and gradually reduces the error, thereby improving the prediction ability of the model. The calculation process of the cross-entropy loss function includes comparing the predicted probability with the true label, calculating the binary cross-entropy loss, and averaging the losses of all samples to obtain the final loss value. The scheme uses the cross-entropy loss function to adaptively learn the model parameters, so that the model can more accurately predict the similarity of asset image pairs, stabilize the training process, and avoid the excessive influence of a single sample on the overall training. Through this optimization process, the scheme significantly improves the effect of asset false match detection, and has good interpretability and easy implementation.

[0093] In some other embodiments, the binary cross entropy loss of the current training data can also be determined based on the actual similarity and predicted similarity of the current training data in the current training data set, and the corresponding weights, and all binary cross entropy losses in the training data set are accumulated and averaged to obtain an average loss; the regularization term Lreg of the asset false match detection model is calculated based on the network parameters and regularization parameter λ of the asset false match detection model; and the final loss is calculated based on the average loss and regularization term of the binary cross entropy loss.

[0094] First, calculate the binary cross entropy loss:

[0095] BCE=-[α·y i ·log(σ(z i ))+β·(1-y i )·log(1-σ(z i ))]

[0096] Among them, α and β are weight parameters of positive samples and negative samples respectively, which are used to adjust the impact of positive and negative samples on the overall loss to deal with the problem of class imbalance. And α+β=1 makes the scale of the total loss consistent, ensuring the stability of training and the interpretability of loss values. Then, the regularization term is calculated.

[0097]

[0098] Among them, λ is the regularization parameter, θj is the model parameter, and finally, the total loss function after combining the binary cross entropy loss and the regularization term is:

[0099]

[0100] Where N is the number of samples, y i is the true label (0 or 1) of the i-th image pair, z i is the predicted output of the model, and σ(zi) is the output of the Sigmoid function.

[0101] This application can effectively adjust the class imbalance problem by introducing weight parameters α and β, so that the model performs better on class imbalanced data sets. By adding the regularization term λ, the model can be restrained from overfitting the training data and the generalization ability of the model can be enhanced. In short, the improved loss function is more stable when processing complex data, reducing the excessive impact of a single sample on the overall training process.

[0102] In some embodiments, in order to prevent gradient explosion, the gradient clipping norm is used to clip the gradient to a preset threshold, and the model parameters θj are updated based on the clipped gradient, and then the regularization term is calculated based on the model parameters, and the total loss is further calculated based on the binary cross entropy loss and the regularization term.

[0103] The image pairs are respectively input into the pre-trained convolutional attention network to train the feature extractor. The network adopts a weight sharing mechanism, and then the extracted features are measured for similarity through a metric function. Finally, the sigmoid function is used to output the asset image similarity score.

[0104] Step S406: model evaluation.

[0105] The performance of the asset false matching detection model is comprehensively analyzed through a series of evaluation methods. The specific methods of model evaluation will be described in detail below and will not be repeated here.

[0106] Example 3

[0107] The present application embodiment provides an asset matching detection method, such as Fig.11 As shown, the method comprises the following steps:

[0108] Step S1102, initializing the similarity calculation model and loading the model weights;

[0109] Step S1104, determine whether the input is a picture or a video. If it is a picture, perform picture normalization processing, use the pre-trained asset false match detection model to predict the similarity of the two images, and output the similarity score.

[0110] Step S1106: If it is a video, perform video normalization, select frame pairs from the video, use the pre-trained asset false match detection model to predict the similarity of the two images, and output the similarity score. Repeat the selection of frame pairs, record the frame with the highest similarity, and output the highest similarity score. Finally, evaluate the matching of the assets based on the similarity score output by the model.

[0111] The training method of the asset false matching detection model is shown above and will not be repeated here.

[0112] Example 4

[0113] The present application also provides a verification method for an asset false matching detection model, such as Fig.12 As shown, the method comprises the following steps:

[0114] Step S1202, setting the verification environment.

[0115] All experiments in this application were completed on the same experimental computer platform, using Python language and based on the Pytorch deep learning framework for experimental development, with the integrated development environment being Pycharm. The specific configuration information of the experimental computer is shown in Table 2.

[0116] Table 2 Experimental environment configuration

[0117]

[0118] Step S1204, collecting data.

[0119] The dataset used in this experiment is a custom dataset collected and produced on major video websites for asset image similarity matching, and the dataset is expanded through a variety of data enhancement techniques. The selection of asset videos needs to cover all details and features of the assets, and capture the overall picture of the assets from different perspectives, which helps to establish a more comprehensive and accurate asset matching model. The asset videos are then converted into frame images, with each category representing an asset, a total of 195 assets, and a total of 9,636 asset images. The dataset samples are as follows: Fig.13 shown.

[0120] Due to the temporal changes and perspective diversity in the asset shooting process, this application adopts an automatic enhancement strategy RandAugment, which achieves data enhancement through simplified parameter adjustment, mainly including two parameters: the number of operations (N) and the transformation amplitude (M). It randomly selects predefined transformation operations and uniformly applies the transformation amplitude to perform combined enhancement. It includes enhancement techniques such as random occlusion, rotation, scaling, flipping, brightness, and contrast adjustment. Table 3 describes the simulation effects of different enhancement techniques.

[0121] Table 3 Description of simulation effect of enhanced technology

[0122]

[0123] Data augmentation technology is used to simulate the shooting of the same asset at different times and from different perspectives, so that the model can match complex images during training and testing to avoid overfitting. Examples of data augmentation include Fig.14 shown.

[0124] Step S1206, setting evaluation criteria.

[0125] First, in the self-built asset dataset, the size of all original images is uniformly set to 105×105 jpg images, and then the images are divided into training set, validation set and test set according to 7:2:1. The weight parameters of the model are trained on the training set, the hyperparameters of the model are adjusted on the validation set, and finally the test is conducted on the test set. The evaluation index of the asset image similarity detection performance is quantitative analysis and the accuracy, precision, recall rate and F1 measurement value are used for evaluation. The images in the test set are divided into similar image pairs and dissimilar image pairs, including a total of 2310 pairs of images. Taking the successful matching of the same asset image from the test set as an example, the matching target asset image is input, and the similarity is greater than 0.85, which is judged to be the same asset; if not, it means that it is not from the same asset. As shown in Table 4, TP (True Positive) is the number of samples whose true label is positive and whose model predicts them to be positive; FP (False Positive) is the number of samples whose true label is negative but whose model predicts them to be positive; TN (True Negative) is the number of samples whose true label is negative and whose model predicts them to be negative; FN (False Negative) is the number of samples whose true label is positive but whose model predicts them to be negative.

[0126] Table 4 Binary classification confusion matrix

[0127]

[0128] The evaluation index calculation formula is as follows:

[0129] 1) The calculation formula of accuracy is shown in formula (8):

[0130]

[0131] 2) The calculation formula of accuracy is shown in formula (9):

[0132]

[0133] 3) The calculation formula of recall rate is shown in formula (10):

[0134]

[0135] 4) The calculation formula of F1 measurement value is shown in formula (11):

[0136]

[0137] Step S1208, training and analysis.

[0138] The model training parameters are shown in Table 5:

[0139] Table 5 Model training parameters

[0140]

[0141] Training process: The batch size is set to 32, the number of iterations is set to 200, the optimizer is selected as Sgd, the cosine learning rate decay is used, the initial learning rate is set to 1e-2, and the minimum learning rate is set to 1e-4. The loss and accuracy change curves during the training process are as follows Fig.15 As shown. Fig.15 It can be seen that the loss leveled off after training to the 150th Epoch, and the model was effectively trained within 200 Epochs.

[0142] In order to remotely manage overseas diversified assets, this application constructs an asset dataset and proposes an asset false matching detection method based on twin neural networks and convolutional attention modules to achieve more accurate and efficient matching audits. Experimental results show that the proposed method significantly improves the accuracy and efficiency of asset similarity calculations. Not only has it achieved remarkable results in asset identification and matching, it has also demonstrated excellent performance in intelligent asset inventory, providing reliable support for asset management.

[0143] In order to ensure the scientificity and rationality of the experiment, this application introduced an ablation test during the experiment. Under the control of variables, the backbone network, improved method and data set processing were adjusted respectively to evaluate the impact of each part on the overall system performance. The experimental results are shown in Table 6:

[0144] Table 6 Comparison of ablation experiment results

[0145]

[0146]

[0147] From the data in Table 6, we can see that after combining the CBAM module and data enhancement technology, the model shows a higher accuracy and F1 value, while maintaining a high precision and recall rate. This shows that the CBAM module and data enhancement have a promoting effect on improving the performance of the model. In addition, ResNet50 introduces residual connections, which helps to alleviate the gradient vanishing problem, making the network easier to train and able to learn more complex feature representations. In the asset matching task, complex feature representation capabilities can better distinguish similar and dissimilar assets.

[0148] Step S1210, comparing similarity matching results.

[0149] Traditional methods for similarity matching of images usually use histograms to extract the color features of images and gray-level co-occurrence matrices to extract the texture features of images, and calculate the similarity of images based on the extracted feature vectors. To further verify the effectiveness of the network, the method of this application is compared with the traditional method. The comparative test is conducted under the same test set, and the performance comparison of different similarity calculation methods in the asset image similarity detection task is shown in Table 7:

[0150] Table 7 Performance comparison of different methods in asset image similarity detection task

[0151]

[0152] Among the traditional methods, the precision of the histogram method is 52.64%, and the recall rate is only 7.76%. Its omission is the most obvious and its asset matching ability is the weakest; the precision of the gray-level co-occurrence matrix method is 51.08%, and the recall rate is 93.25%. Although the matching ability is strong, it is prone to misjudgment and misjudges many dissimilar assets as similar assets. Therefore, both methods cannot meet the actual application scenarios. In contrast, the accuracy of the method proposed in this application reaches 92.42%, which is 39.78% and 41.34% higher than the histogram method and the gray-level co-occurrence matrix method respectively; the F1 measure is 92.48%, which is 78.08% and 26.31% higher than the two traditional methods respectively. These results show that the asset similarity matching method based on deep learning is significantly better than the traditional method in performance.

[0153] Example 5

[0154] The present application also provides a training device for an asset false matching detection model based on deep learning, such as Fig.16As shown, it includes: a construction module 192, which is configured to fuse the convolutional neural network and the twin network combined with the attention mechanism to construct the asset false match detection model; a feature extraction module 194, which is configured to input the training data into the convolutional neural network, and use the convolutional neural network to extract features of two sample images forming positive samples or negative samples in the training data to obtain two feature vectors; a similarity comparison module, which is configured to compare the feature differences between the two feature vectors, and predict the similarity of the two sample images based on the feature differences; a training module 196, which is configured to compare the predicted similarity with the actual similarity, obtain the loss between the predicted similarity and the actual similarity, and update the network parameters of the convolutional neural network and the twin network based on the loss to obtain the trained asset false match detection model.

[0155] It should be noted that the training device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the training device provided in the above embodiment belongs to the same concept as the training method embodiment. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0156] Example 6

[0157] Fig.17 FIG. 1 shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present disclosure. It should be noted that: Fig.17 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0158] like Fig.17 As shown, the electronic device includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for system operation are also stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0159] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0160] In particular, according to an embodiment of the present disclosure, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, various functions defined in the method and apparatus of the present application are executed. In some embodiments, the electronic device may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0161] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0162] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0163] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0164] As another aspect, the present application further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device.

[0165] The computer-readable medium carries one or more programs, and when the one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments. For example, the electronic device can implement each step of the method embodiment.

[0166] As another aspect, the present application further provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program executable by the at least one processor, and the computer program is used to enable the electronic device to perform the method of the embodiment of the present application when executed by the at least one processor.

[0167] As another aspect, the present application further provides a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiment of the present application.

[0168] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0169] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0170] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0171] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0172] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0173] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0174] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An asset matching detection method, characterized in that: include: Initializing an asset false match detection model and loading the weights of the asset false match detection model; In the case where the asset data input to the asset false match detection model is two pictures, normalizing the two pictures respectively, and using the asset false match detection model to perform similarity prediction on the two pictures, and outputting the predicted similarity score; In the case where the asset data input to the asset false match detection model is two videos, normalizing the two videos respectively, selecting frame pairs from the two normalized videos to obtain two pictures corresponding to the frame pairs, and using the asset false match detection model to perform similarity prediction on the two pictures, and outputting the predicted similarity scores; Based on the similarity score, it is determined whether the asset data matches.

2. The method according to claim 1, characterized in that The asset false matching detection model is obtained by using the twin neural network as the basic framework to construct a convolutional neural network and introduce a hybrid attention mechanism.

3. The method according to claim 2, characterized in that Using the asset false match detection model to predict the similarity of the two images, and outputting the predicted similarity scores, including: Using the convolutional neural network to extract features from the two images; The extracted features are measured for similarity using the metric function of the twin neural network, and the predicted similarity score is output.

4. The method according to claim 3, characterized in that: The similarity measurement of the extracted features is performed using the metric function of the twin neural network, including: Calculating the distance between two feature vectors corresponding to the extracted features respectively using the metric function; The similarity measurement is performed based on the distance.

5. The method according to claim 3, characterized in that: Using the convolutional neural network to extract features from the two images includes: Scan the two images using multiple filters to generate feature maps, and reduce the size of the feature maps by downsampling; An activation function is used to introduce nonlinear features into the feature map with a reduced size, and the feature map with the introduced nonlinear features is used as the extracted feature.

6. The method according to claim 5, characterized in that After generating the feature map, the method further includes: Using the channel attention module, the importance of each channel is captured through global average pooling and global maximum pooling, and the channel weights are generated through a multi-layer perceptron; Using a spatial attention module, the feature map is pooled in the channel dimension, and a spatial weight is generated through a convolution operation; The weight of the feature map is dynamically adjusted based on the channel weight and the spatial weight.

7. An asset matching detection device, characterized in that: include: An initialization module, configured to initialize the asset false match detection model and load the weight of the asset false match detection model; The detection module is configured to: when the asset data input into the asset false match detection model is two pictures, respectively normalize the two pictures, use the asset false match detection model to perform similarity prediction on the two pictures, and output a predicted similarity score; In the case where the asset data input to the asset false match detection model is two videos, normalizing the two videos respectively, selecting frame pairs from the two normalized videos to obtain two pictures corresponding to the frame pairs, and using the asset false match detection model to perform similarity prediction on the two pictures, and outputting the predicted similarity scores; The determination module is configured to determine whether the asset data matches based on the similarity score.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 6.

9. A computer device, characterized in that: include: Memory and processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor is enabled to execute the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images