A multi-modal item value fitting evaluation method and system

Abstract

The present invention discloses a multimodal-based item value fitting evaluation method and system, which relates to the field of computer vision technology, including: collecting multimodal data of items to be detected, and a multimodal classification sample data set of items to be detected; performing image-text feature extraction and splicing on the multimodal classification sample data set of items to be detected, and obtaining a positive and negative sample matrix of the image-text feature vector of the items to be detected; performing network item matching screening based on the image-text feature vector sample matrix of the items to be detected, and obtaining a network reference item value sample set of the items to be detected; constructing a multimodal item value detection model based on a Transformer architecture; training a multimodal item value detection model based on the network reference item value sample set of the items to be detected, substituting the image-text feature vector sample matrix of the items to be detected into the fitting evaluation, and generating the value of the items to be detected. The present invention has the advantages of improving the accuracy and efficiency of item value evaluation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a multi-modal based object value fitting evaluation method and system. Background Art

[0002] Item valuation refers to the process of estimating and determining the economic value of a specific item at a certain point in time through scientific methods and procedures, which aims to reflect the reasonable price or value of the item in scenarios such as market transactions, insurance claims, tax declarations, asset liquidations or investment decisions.

[0003] Since the existing item value assessment system is subject to interference from human subjective factors, there is a large gap between the value assessment of items and their actual value, and when applied to third-party price testing, the fairness of the assessed value cannot be considered. Summary of the invention

[0004] In order to solve the above technical problems, a multimodal-based item value fitting evaluation method and system are provided. This technical solution solves the above-mentioned problem that the existing item value evaluation system is interfered by human subjective factors, the value evaluation of the item is quite different from the actual value, and when it is applied to third-party price detection, the fairness of the evaluation value cannot be taken into consideration.

[0005] In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0006] A multi-modal item value fitting evaluation method, comprising:

[0007] Collect multimodal data of the object to be detected, perform initial data division according to category rules, and form a multimodal classification sample data set of the object to be detected; the category rules include: image data, text data;

[0008] Perform image and text feature extraction and splicing on the multimodal classification sample data set of the object to be detected to obtain the positive and negative sample matrix of the image and text feature vector of the object to be detected;

[0009] Perform network item matching screening based on the image and text feature vector sample matrix of the item to be detected, and obtain a network reference item value sample set of the item to be detected;

[0010] Based on the Transformer architecture, a multimodal item value detection model is built;

[0011] Based on the network reference item value sample set of the items to be detected, a multimodal item value detection model is trained, and the sample matrix of the graphic feature vectors of the items to be detected is substituted into the fitting evaluation to generate the value of the items to be detected.

[0012] Preferably, performing image-text feature extraction and splicing on a multimodal classification sample data set of an object to be detected to obtain a positive and negative sample matrix of an image-text feature vector of the object to be detected specifically includes:

[0013] Based on the multimodal classification sample data set of the objects to be detected, the image feature parameters and text feature parameters of the objects to be detected are extracted according to the classification type, and a number index is marked for each image feature parameter and text feature parameter;

[0014] Based on the alignment of each image feature parameter and text feature parameter according to the marked number index, the image feature parameters and text feature parameters of the object to be detected with the same number index are spliced ​​into a graphic feature vector through the average weighted method, and combined into a graphic feature vector sample set of the object to be detected;

[0015] Calculate the cosine similarity value of each pair of image-text feature vector samples in the image-text feature vector sample set of the object to be detected, and recursively traverse each pair of image-text feature vector samples to select the maximum similarity value of each pair of image-text feature vector samples as the positive sample of the image-text feature vector of the object to be detected, and select the minimum similarity value of each pair of image-text feature vector samples as the negative sample of the image-text feature vector of the object to be detected;

[0016] Combine the positive sample of the image-text feature vector of the object to be detected and the negative sample of the image-text feature vector of the object to be detected into a positive and negative sample matrix of the image-text feature vector of the object to be detected;

[0017] The positive and negative sample matrix of the image feature vector of the object to be detected is specifically:

[0018] ;

[0019] Where B is the positive and negative sample matrix of the image feature vector of the object to be detected, b ij are the positive and negative samples of the image and text feature vectors of the object to be detected, n is the total number of image feature vectors, and m is the total number of text feature vectors;

[0020] The average weighted method is specifically:

[0021] ;

[0022] Where A is the sample set of image and text feature vectors of the object to be detected, is the i-th image feature parameter weight vector indexed by the k-th number of the detected object, is the i-th image feature parameter of the k-th number index of the detected object, is the jth text feature parameter weight vector indexed by the kth number of the detected object, The jth text feature parameter indexed by the kth number of the detected object;

[0023] The cosine similarity value of each pair of image-text feature vector samples in the image-text feature vector sample set of the object to be detected is calculated as follows:

[0024] ;

[0025] Where P ij is the similarity value between the image and text feature vector samples of the object to be detected, is the i-th image feature vector with the k-th number index in the image-text feature vector sample set, is the j-th text feature vector indexed by the k-th number in the sample set of image and text feature vectors.

[0026] Preferably, performing network item search, matching and screening based on the positive and negative sample matrix of the image and text feature vector of the item to be detected to obtain a network reference item value sample set of the item to be detected specifically includes:

[0027] Based on the API interface of the second-hand trading platform, obtain historical transaction big data;

[0028] Based on the historical transaction big data, the transaction items and transaction values ​​in the historical transaction big data are linearly mapped according to the timestamps to obtain the sample vector matrix of the network transaction item value;

[0029] ;

[0030] Where D is the sample vector matrix of the value of online transaction items, d ct is the transaction value of the cth transaction item at the tth transaction time, v is the total number of transaction items, and s is the total time length;

[0031] Recursively match each element in the positive and negative sample matrix of the image and text feature vector of the object to be detected with the network transaction item value sample vector matrix, and select the network transaction item value sample vector corresponding to the maximum value of the positive and negative sample matching degree of the image and text feature vector of the detected object in the network transaction item value sample vector matrix, and form a network reference item value sample set of the object to be detected;

[0032] Among them, the network reference item value sample set of the items to be detected is specifically:

[0033] ;

[0034] Among them, S is the network reference item value sample set of the item to be detected, f(b k' ,d k') is the matching value between the positive and negative samples of the image feature vector of the k'th index to be detected and the value sample vector of the k'th index network transaction item, G k' It is the minimum distance between the positive and negative samples of the image feature vector of the k'th index to be detected and the value sample vector of the k'th index network item.

[0035] Preferably, based on the network reference item value sample set of the item to be detected, a multimodal item value detection model is trained, and the image and text feature vector sample matrix of the item to be detected is substituted into the fitting evaluation to generate the value of the item to be detected, which specifically includes:

[0036] Based on the network reference item value sample set of the object to be detected, weights are assigned to the features of each network reference item value sample of the object to be detected by the expectation maximization formula to obtain a weighted feature set of the network reference item value sample of the object to be detected; the features include: image appearance features, image defect features, and text description features;

[0037] Based on the weighted feature set of the network reference item value samples of the items to be detected, the weighted features of the network reference item value samples of each item to be detected are substituted into the multimodal item value detection model, the actual value parameters of the network reference item value samples of each item to be detected are used as output, and the minimum error function of the multimodal item value detection model is used as the training end target to obtain the trained multimodal item value detection model;

[0038] Based on the multimodal classification sample data set of the objects to be detected, the trained multimodal object value detection model is input to generate the evaluation value parameters of the objects to be detected;

[0039] The expectation maximization formula is specifically:

[0040] ;

[0041] In the formula, w ct is the weighted weight of the tth transaction time of the cth transaction item in the network reference item value sample of the item to be detected; r(S|θ) is the conditional probability of the set S of network reference item value samples of the item to be detected under the given parameter θ of the logarithmic function; I' is the total number of indexes of the network reference item value samples of the item to be detected.

[0042] Furthermore, a multimodal-based item value fitting evaluation system is proposed, which is used to implement the multimodal-based item value fitting evaluation method as described above, including:

[0043] The data acquisition module is used to collect multimodal data of the objects to be detected, perform initial data division according to the classification rules, and form a multimodal classification sample data set of the objects to be detected;

[0044] A feature fusion module, which is electrically connected to the data acquisition module, and is used to extract and splice image and text features of a multimodal classification sample data set of an object to be detected, and obtain a positive and negative sample matrix of an image and text feature vector of the object to be detected;

[0045] A network search and matching module, the network search and matching module is electrically connected to the feature fusion module, and the network search and matching module is used to perform network item matching screening based on the image and text feature vector sample matrix of the item to be detected, and obtain a network reference item value sample set of the item to be detected;

[0046] Model building module: The model building module is used to build a multimodal item value detection model based on the Transformer architecture;

[0047] The value assessment module is electrically connected to the network search and matching module and the model building module. The value assessment module is used to train a multimodal item value detection model based on a network reference item value sample set of the item to be detected, substitute the graphic feature vector sample matrix of the item to be detected for fitting evaluation, and generate the value of the item to be detected.

[0048] Optionally, the feature fusion module includes:

[0049] The classification unit extracts the image feature parameters and text feature parameters of the object to be detected based on the multimodal classification sample data set of the object to be detected according to the classification type, and marks a number index for each image feature parameter and text feature parameter;

[0050] The feature concatenation unit aligns each image feature parameter and text feature parameter according to the marked number index, and concatenates the image feature parameters and text feature parameters with the same number index of the object to be detected into a graphic feature vector by an average weighted method, and combines them into a graphic feature vector sample set of the object to be detected;

[0051] The positive and negative sample unit calculates the cosine similarity value of each pair of image-text feature vector samples in the image-text feature vector sample set of the object to be detected, and recursively traverses each pair of image-text feature vector samples to select the maximum similarity value of each pair of image-text feature vector samples as the positive sample of the image-text feature vector of the object to be detected, and selects the minimum similarity value of each pair of image-text feature vector samples as the negative sample of the image-text feature vector of the object to be detected;

[0052] The positive and negative sample matrix unit combines the positive sample of the image-text feature vector of the object to be detected and the negative sample of the image-text feature vector of the object to be detected into a positive and negative sample matrix of the image-text feature vector of the object to be detected.

[0053] Optionally, the network search matching module includes:

[0054] The transaction big data unit obtains historical transaction big data based on the API interface of the second-hand transaction platform;

[0055] The transaction vector matrix unit, based on the historical transaction big data, performs linear mapping between the transaction items and transaction values ​​in the historical transaction big data according to the timestamp, and obtains the sample vector matrix of the network transaction item value;

[0056] Referring to the sample set unit, each element in the positive and negative sample matrix of the image and text feature vector of the object to be detected is recursively matched with the network transaction item value sample vector matrix, and the network transaction item value sample vector corresponding to the maximum value of the positive and negative sample matching degree of the image and text feature vector of the detected object in the network transaction item value sample vector matrix is ​​screened out to form a network reference item value sample set for the object to be detected.

[0057] Optionally, the value assessment module includes:

[0058] A weighting unit, based on the network reference item value sample set of the item to be detected, assigns a weight to the feature of each network reference item value sample of the item to be detected by using an expectation maximization formula, and obtains a weighted feature set of the network reference item value sample of the item to be detected;

[0059] The model training unit, based on the weighted feature set of the network reference item value samples of the items to be detected, substitutes the weighted features of the network reference item value samples of each item to be detected into the multimodal item value detection model, takes the actual value parameters of the network reference item value samples of each item to be detected as output, takes the minimum error function of the multimodal item value detection model as the training end target, and obtains the trained multimodal item value detection model;

[0060] The item value assessment unit, based on the multimodal classification sample data set of the item to be detected, inputs the trained multimodal item value detection model to generate the assessment value parameters of the item to be detected.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] The present invention proposes a multimodal item value fitting evaluation scheme, which collects multimodal (image and text) data of the items to be detected, and initially divides the data according to the category rules to construct a classification sample data set, and extracts and splices the image and text features to form a feature vector sample matrix of the items to be detected. Using this matrix, the value sample set of similar items is matched and screened in the network. The multimodal item value detection model built based on the Transformer architecture improves the accuracy and efficiency of item value evaluation by training the model and using the image and text feature vector sample matrix for fitting evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A flowchart of a multi-modal-based item value fitting assessment method;

[0064] Figure 2 Flow chart of the method for obtaining the positive and negative sample matrix of the image feature vector of the object to be detected;

[0065] Figure 3 A flow chart of a method for obtaining a network reference item value sample set for an item to be detected;

[0066] Figure 4 A flow chart of a method for generating the value of an item to be tested;

[0067] Figure 5 This is a framework diagram of a multimodal item value fitting evaluation system. DETAILED DESCRIPTION

[0068] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art may think of other obvious variations.

[0069] Reference Figure 1 As shown, a multi-modal item value fitting evaluation method includes:

[0070] Collect multimodal data of the object to be detected, perform initial data division according to category rules, and form a multimodal classification sample data set of the object to be detected; the category rules include: image data, text data;

[0071] Perform image and text feature extraction and splicing on the multimodal classification sample data set of the object to be detected to obtain the positive and negative sample matrix of the image and text feature vector of the object to be detected;

[0072] Perform network item matching screening based on the image and text feature vector sample matrix of the item to be detected, and obtain a network reference item value sample set of the item to be detected;

[0073] Based on the Transformer architecture, a multimodal item value detection model is built;

[0074] Based on the network reference item value sample set of the items to be detected, a multimodal item value detection model is trained, and the sample matrix of the graphic feature vectors of the items to be detected is substituted into the fitting evaluation to generate the value of the items to be detected.

[0075] This solution collects multimodal (image and text) data of the items to be detected, and initially divides the data according to the classification rules to construct a classification sample data set. By extracting and splicing the image and text features, a feature vector sample matrix of the items to be detected is formed. Using this matrix, the value sample set of similar items is matched and screened in the network. The multimodal item value detection model built based on the Transformer architecture improves the accuracy and efficiency of item value assessment by training the model and using the image and text feature vector sample matrix for fitting assessment.

[0076] Reference Figure 2 As shown, the image and text feature extraction and splicing are performed on the multimodal classification sample data set of the object to be detected, and the positive and negative sample matrices of the image and text feature vectors of the object to be detected are obtained, which specifically includes:

[0077] Based on the multimodal classification sample data set of the objects to be detected, the image feature parameters and text feature parameters of the objects to be detected are extracted according to the classification type, and a number index is marked for each image feature parameter and text feature parameter;

[0078] Based on the alignment of each image feature parameter and text feature parameter according to the marked number index, the image feature parameters and text feature parameters of the object to be detected with the same number index are spliced ​​into a graphic feature vector through the average weighted method, and combined into a graphic feature vector sample set of the object to be detected;

[0079] Calculate the cosine similarity value of each pair of image-text feature vector samples in the image-text feature vector sample set of the object to be detected, and recursively traverse each pair of image-text feature vector samples to select the maximum similarity value of each pair of image-text feature vector samples as the positive sample of the image-text feature vector of the object to be detected, and select the minimum similarity value of each pair of image-text feature vector samples as the negative sample of the image-text feature vector of the object to be detected;

[0080] Combine the positive sample of the image-text feature vector of the object to be detected and the negative sample of the image-text feature vector of the object to be detected into a positive and negative sample matrix of the image-text feature vector of the object to be detected;

[0081] The positive and negative sample matrix of the image feature vector of the object to be detected is specifically:

[0082] ;

[0083] Where B is the positive and negative sample matrix of the image feature vector of the object to be detected, b ij are the positive and negative samples of the image and text feature vectors of the object to be detected, n is the total number of image feature vectors, and m is the total number of text feature vectors;

[0084] The average weighted method is specifically:

[0085] ;

[0086] Where A is the sample set of image and text feature vectors of the object to be detected, is the i-th image feature parameter weight vector indexed by the k-th number of the detected object, is the i-th image feature parameter of the k-th number index of the detected object, is the jth text feature parameter weight vector indexed by the kth number of the detected object, The jth text feature parameter indexed by the kth number of the detected object;

[0087] The cosine similarity value of each pair of image-text feature vector samples in the image-text feature vector sample set of the object to be detected is calculated as follows:

[0088] ;

[0089] Where P ij is the similarity value between the image and text feature vector samples of the object to be detected, is the i-th image feature vector with the k-th number index in the image-text feature vector sample set, is the j-th text feature vector indexed by the k-th number in the sample set of image and text feature vectors.

[0090] It should be noted that since the image features and text features of the object to be detected will have inconsistencies in positive and negative samples when the positive and negative samples are screened after feature splicing, there will be fitting fluctuations in the subsequent model training and recognition process. Therefore, after the maximum and minimum values ​​of the similarity of each group of image and text feature vectors are screened, the consistency of the training samples is guaranteed to improve the accuracy of model recognition.

[0091] Reference Figure 3 As shown, based on the positive and negative sample matrix of the image and text feature vector of the object to be detected, the network item search matching screening is performed to obtain the network reference item value sample set of the object to be detected, which specifically includes:

[0092] Based on the API interface of the second-hand trading platform, obtain historical transaction big data;

[0093] Based on the historical transaction big data, the transaction items and transaction values ​​in the historical transaction big data are linearly mapped according to the timestamps to obtain the sample vector matrix of the network transaction item value;

[0094] ;

[0095] Where D is the sample vector matrix of the value of online transaction items, d ct is the transaction value of the cth transaction item at the tth transaction time, v is the total number of transaction items, and s is the total time length;

[0096] Recursively match each element in the positive and negative sample matrix of the image and text feature vector of the object to be detected with the network transaction item value sample vector matrix, and select the network transaction item value sample vector corresponding to the maximum value of the positive and negative sample matching degree of the image and text feature vector of the detected object in the network transaction item value sample vector matrix, and form a network reference item value sample set of the object to be detected;

[0097] Among them, the network reference item value sample set of the items to be detected is specifically:

[0098] ;

[0099] Among them, S is the network reference item value sample set of the item to be detected, f(b k' ,d k' ) is the matching value between the positive and negative samples of the image feature vector of the k'th index to be detected and the value sample vector of the k'th index network transaction item, G k' It is the minimum distance between the positive and negative samples of the image feature vector of the k'th index to be detected and the value sample vector of the k'th index network item.

[0100] This solution obtains historical transaction big data through the API interface based on the second-hand transaction platform, and linearly maps these data to generate a network transaction item value sample vector matrix, recursively matches each element in the positive and negative sample matrix of the image and text feature vector of the item to be detected with the network transaction item value sample vector matrix, and selects the network transaction item value sample vector corresponding to the maximum value of the positive and negative sample matching degree of the image and text feature vector of the item to be detected, thereby constructing a network reference item value sample set of the item to be detected. This solution provides a value reference for subsequent decision-making by accurately matching the value samples of network transaction items.

[0101] Reference Figure 4 As shown, based on the network reference item value sample set of the items to be detected, the multimodal item value detection model is trained, and the image and text feature vector sample matrix of the items to be detected is substituted into the fitting evaluation to generate the value of the items to be detected, which specifically includes:

[0102] Based on the network reference item value sample set of the object to be detected, weights are assigned to the features of each network reference item value sample of the object to be detected by the expectation maximization formula to obtain a weighted feature set of the network reference item value sample of the object to be detected; the features include: image appearance features, image defect features, and text description features;

[0103] Based on the weighted feature set of the network reference item value samples of the items to be detected, the weighted features of the network reference item value samples of each item to be detected are substituted into the multimodal item value detection model, the actual value parameters of the network reference item value samples of each item to be detected are used as output, and the minimum error function of the multimodal item value detection model is used as the training end target to obtain the trained multimodal item value detection model;

[0104] Based on the multimodal classification sample data set of the objects to be detected, the trained multimodal object value detection model is input to generate the evaluation value parameters of the objects to be detected;

[0105] The expectation maximization formula is specifically:

[0106] ;

[0107] In the formula, w ct is the weighted weight of the tth transaction time of the cth transaction item in the network reference item value sample of the item to be detected; r(S|θ) is the conditional probability of the set S of network reference item value samples of the item to be detected under the given parameter θ of the logarithmic function; I' is the total number of indexes of the network reference item value samples of the item to be detected.

[0108] It should be noted that since the transaction value on the trading platform itself represents the degree of acceptance of the item by the target audience, the transaction value of the transaction items obtained in this solution is the inventory and residual value range of the items accepted by the general public in reality, ensuring that the item value load evaluated by the trained multimodal item value detection model is within the range of recognition by the target audience;

[0109] Reference Figure 5 As shown, based on the same inventive concept of a multi-modal item value fitting evaluation method, a multi-modal item value fitting evaluation system is proposed, comprising:

[0110] The data acquisition module is used to collect multimodal data of the objects to be detected, perform initial data division according to the classification rules, and form a multimodal classification sample data set of the objects to be detected;

[0111] A feature fusion module, which is electrically connected to the data acquisition module, and is used to extract and splice image and text features of a multimodal classification sample data set of an object to be detected, and obtain a positive and negative sample matrix of an image and text feature vector of the object to be detected;

[0112] A network search and matching module, the network search and matching module is electrically connected to the feature fusion module, and the network search and matching module is used to perform network item matching screening based on the image and text feature vector sample matrix of the item to be detected, and obtain a network reference item value sample set of the item to be detected;

[0113] Model building module: The model building module is used to build a multimodal item value detection model based on the Transformer architecture;

[0114] The value assessment module is electrically connected to the network search and matching module and the model building module. The value assessment module is used to train a multimodal item value detection model based on a network reference item value sample set of the item to be detected, substitute the graphic feature vector sample matrix of the item to be detected for fitting evaluation, and generate the value of the item to be detected.

[0115] Among them, the feature fusion module includes:

[0116] The classification unit extracts the image feature parameters and text feature parameters of the object to be detected based on the multimodal classification sample data set of the object to be detected according to the classification type, and marks a number index for each image feature parameter and text feature parameter;

[0117] The feature concatenation unit aligns each image feature parameter and text feature parameter according to the marked number index, and concatenates the image feature parameters and text feature parameters with the same number index of the object to be detected into a graphic feature vector by an average weighted method, and combines them into a graphic feature vector sample set of the object to be detected;

[0118] The positive and negative sample unit calculates the cosine similarity value of each pair of image-text feature vector samples in the image-text feature vector sample set of the object to be detected, and recursively traverses each pair of image-text feature vector samples to select the maximum similarity value of each pair of image-text feature vector samples as the positive sample of the image-text feature vector of the object to be detected, and selects the minimum similarity value of each pair of image-text feature vector samples as the negative sample of the image-text feature vector of the object to be detected;

[0119] The positive and negative sample matrix unit combines the positive sample of the image-text feature vector of the object to be detected and the negative sample of the image-text feature vector of the object to be detected into a positive and negative sample matrix of the image-text feature vector of the object to be detected.

[0120] Among them, the network search and matching module includes:

[0121] The transaction big data unit obtains historical transaction big data based on the API interface of the second-hand transaction platform;

[0122] The transaction vector matrix unit, based on the historical transaction big data, performs linear mapping between the transaction items and transaction values ​​in the historical transaction big data according to the timestamp, and obtains the sample vector matrix of the network transaction item value;

[0123] Referring to the sample set unit, each element in the positive and negative sample matrix of the image and text feature vector of the object to be detected is recursively matched with the network transaction item value sample vector matrix, and the network transaction item value sample vector corresponding to the maximum value of the positive and negative sample matching degree of the image and text feature vector of the detected object in the network transaction item value sample vector matrix is ​​screened out to form a network reference item value sample set for the object to be detected.

[0124] The value assessment module includes:

[0125] A weighting unit, based on the network reference item value sample set of the item to be detected, assigns a weight to the feature of each network reference item value sample of the item to be detected by using an expectation maximization formula, and obtains a weighted feature set of the network reference item value sample of the item to be detected;

[0126] The model training unit, based on the weighted feature set of the network reference item value samples of the items to be detected, substitutes the weighted features of the network reference item value samples of each item to be detected into the multimodal item value detection model, takes the actual value parameters of the network reference item value samples of each item to be detected as output, takes the minimum error function of the multimodal item value detection model as the training end target, and obtains the trained multimodal item value detection model;

[0127] The item value assessment unit, based on the multimodal classification sample data set of the item to be detected, inputs the trained multimodal item value detection model to generate the assessment value parameters of the item to be detected.

[0128] The usage process of a multi-modal item value fitting evaluation system is as follows:

[0129] Step 1: Collect multimodal data of the objects to be detected, perform initial data division according to classification rules, and build a multimodal classification sample data set of the objects to be detected;

[0130] Step 2: Based on the multimodal classification sample data set of the object to be detected, extract the image feature parameters and text feature parameters of the object to be detected according to the classification type, and mark a number index for each image feature parameter and text feature parameter;

[0131] Step 3: Based on the alignment of each image feature parameter and text feature parameter according to the marked number index, the image feature parameters and text feature parameters of the object to be detected with the same number index are spliced ​​into a graphic feature vector through the average weighted method, and combined into a graphic feature vector sample set of the object to be detected;

[0132] Step 4: Calculate the cosine similarity value of each pair of image-text feature vector samples in the image-text feature vector sample set of the object to be detected, and recursively traverse each pair of image-text feature vector samples to select the maximum similarity value of each pair of image-text feature vector samples as the positive sample of the image-text feature vector of the object to be detected, and select the minimum similarity value of each pair of image-text feature vector samples as the negative sample of the image-text feature vector of the object to be detected;

[0133] Step 5: Combine the positive sample of the image-text feature vector of the object to be detected and the negative sample of the image-text feature vector of the object to be detected into a positive and negative sample matrix of the image-text feature vector of the object to be detected;

[0134] Step 6: Obtain historical transaction big data based on the API interface of the trading platform;

[0135] Step 7: Based on the historical transaction big data, perform linear mapping between the transaction items and transaction values ​​in the historical transaction big data according to the timestamps to obtain a sample vector matrix of the value of network transaction items;

[0136] Step 8: recursively match each element in the positive and negative sample matrix of the image and text feature vector of the object to be detected with the network transaction item value sample vector matrix, and select the network transaction item value sample vector corresponding to the maximum value of the positive and negative sample matching degree of the image and text feature vector of the detected object in the network transaction item value sample vector matrix, and form a network reference item value sample set for the object to be detected;

[0137] Step 9: Build a multimodal item value detection model based on the Transformer architecture;

[0138] Step 10: Based on the network reference item value sample set of the item to be detected, weights are assigned to the features of each network reference item value sample of the item to be detected by using the expectation maximization formula to obtain a weighted feature set of the network reference item value samples of the item to be detected;

[0139] Step 11: Based on the weighted feature set of the network reference item value samples of the items to be detected, the weighted features of the network reference item value samples of each item to be detected are substituted into the multimodal item value detection model, the actual value parameters of the network reference item value samples of each item to be detected are used as output, and the error function of the minimum multimodal item value detection model is used as the training end target to obtain the trained multimodal item value detection model;

[0140] Step 12: Based on the multimodal classification sample data set of the items to be detected, the trained multimodal item value detection model is input to generate the evaluation value parameters of the items to be detected.

[0141] In summary, the advantages of the present invention are: by collecting multimodal (image and text) data of the object to be detected, and initially dividing the data according to the category rules to construct a classification sample data set, by extracting and splicing the image and text features, a feature vector sample matrix of the object to be detected is formed. Using this matrix, the value sample set of similar objects is matched and screened in the network. The multimodal object value detection model built based on the Transformer architecture improves the accuracy and efficiency of the object value assessment by training the model and using the image and text feature vector sample matrix for fitting assessment.

[0142] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A multi-modal item value fitting evaluation method, It is characterized in that include: Collect multimodal data of the objects to be detected, perform initial data division according to classification rules, and build a multimodal classification sample data set of the objects to be detected; The category rules include: image data, text data; Perform image and text feature extraction and splicing on the multimodal classification sample data set of the object to be detected to obtain the positive and negative sample matrix of the image and text feature vector of the object to be detected; Perform network item matching screening based on the image and text feature vector sample matrix of the item to be detected, and obtain a network reference item value sample set of the item to be detected; Based on the Transformer architecture, a multimodal item value detection model is built; Based on the network reference item value sample set of the items to be detected, a multimodal item value detection model is trained, and the image and text feature vector sample matrix of the items to be detected is substituted into the fitting evaluation to generate the value of the items to be detected; The extraction and splicing of image and text features of the multimodal classification sample data set for the object to be detected to obtain the positive and negative sample matrices of the image and text feature vectors of the object to be detected specifically includes: Based on the multimodal classification sample data set of the object to be detected, the image feature parameters and text feature parameters of the object to be detected are extracted according to the classification type, and a number index is marked for each image feature parameter and text feature parameter; Based on the alignment of each image feature parameter and text feature parameter according to the marked number index, the image feature parameters and text feature parameters of the object to be detected with the same number index are spliced ​​into a graphic feature vector through the average weighted method, and combined into a graphic feature vector sample set of the object to be detected; Calculate the cosine similarity value of each pair of image-text feature vector samples in the image-text feature vector sample set of the object to be detected, and recursively traverse each pair of image-text feature vector samples to select the maximum similarity value of each pair of image-text feature vector samples as the positive sample of the image-text feature vector of the object to be detected, and select the minimum similarity value of each pair of image-text feature vector samples as the negative sample of the image-text feature vector of the object to be detected; Combine the positive sample of the image-text feature vector of the object to be detected and the negative sample of the image-text feature vector of the object to be detected into a positive and negative sample matrix of the image-text feature vector of the object to be detected; The method of performing network item search, matching and screening based on the positive and negative sample matrix of the image and text feature vector of the object to be detected to obtain a network reference item value sample set of the object to be detected specifically includes: Based on the API interface of the second-hand trading platform, obtain historical transaction big data; Based on the historical transaction big data, the transaction items and transaction values ​​in the historical transaction big data are linearly mapped according to the timestamps to obtain the sample vector matrix of the network transaction item value; ； Where D is the sample vector matrix of the value of online transaction items, d ct is the transaction value of the cth transaction item at the tth transaction time, v is the total number of transaction items, and s is the total time length; Recursively match each element in the positive and negative sample matrix of the image and text feature vector of the object to be detected with the network transaction item value sample vector matrix, and select the network transaction item value sample vector corresponding to the maximum value of the positive and negative sample matching degree of the image and text feature vector of the detected object in the network transaction item value sample vector matrix, and form a network reference item value sample set of the object to be detected; Among them, the network reference item value sample set of the items to be detected is specifically: ; Among them, S is the network reference item value sample set of the item to be detected, f(b k' d k' ) is the matching value between the positive and negative samples of the image feature vector of the k'th index to be detected and the value sample vector of the k'th index network transaction item, G k' It is the minimum distance between the positive and negative samples of the image feature vector of the k'th index to be detected and the value sample vector of the k'th index network item.

2. According to the multi-modal item value fitting evaluation method of claim 1, It is characterized in that The positive and negative sample matrix of the image and text feature vector of the object to be detected is specifically: ; Where B is the positive and negative sample matrix of the image feature vector of the object to be detected, b ij are the positive and negative samples of the image and text feature vectors of the object to be detected, n is the total number of image feature vectors, and m is the total number of text feature vectors; The average weighted method is specifically: ; Where A is the sample set of image and text feature vectors of the object to be detected, is the i-th image feature parameter weight vector indexed by the k-th number of the detected object, is the i-th image feature parameter of the k-th number index of the detected object, is the jth text feature parameter weight vector indexed by the kth number of the detected object, The jth text feature parameter indexed by the kth number of the detected object; The cosine similarity value of each pair of image-text feature vector samples in the image-text feature vector sample set of the object to be detected is calculated as follows: ; Where P ij is the similarity value between the image and text feature vector samples of the object to be detected, is the i-th image feature vector with the k-th number index in the image-text feature vector sample set, is the j text feature vector indexed by the kth number in the sample set of image and text feature vectors.

3. According to the multi-modal item value fitting evaluation method of claim 2, It is characterized in that Based on the network reference item value sample set of the items to be detected, a multimodal item value detection model is trained, and the image and text feature vector sample matrix of the items to be detected is substituted into the fitting evaluation to generate the value of the items to be detected, which specifically includes: Based on the network reference item value sample set of the object to be detected, weights are assigned to the features of each network reference item value sample of the object to be detected by the expectation maximization formula to obtain a weighted feature set of the network reference item value sample of the object to be detected; the features include: image appearance features, image defect features, and text description features; Based on the weighted feature set of the network reference item value samples of the items to be detected, the weighted features of the network reference item value samples of each item to be detected are substituted into the multimodal item value detection model, the actual value parameters of the network reference item value samples of each item to be detected are used as output, and the minimum error function of the multimodal item value detection model is used as the training end target to obtain the trained multimodal item value detection model; Based on the multimodal classification sample data set of the objects to be detected, the trained multimodal object value detection model is input to generate the evaluation value parameters of the objects to be detected; The expectation maximization formula is specifically: ; In the formula, w ct is the weighted weight of the tth transaction time of the cth transaction item in the network reference item value sample of the item to be detected; r(S|θ) is the conditional probability of the set S of network reference item value samples of the item to be detected under the given parameter θ of the logarithmic function; I' is the total number of indexes of the network reference item value samples of the item to be detected.

4. A multi-modal item value fitting evaluation system, It is characterized in that A method for implementing a multimodal item value fitting evaluation method as claimed in any one of claims 1 to 3, comprising: The data acquisition module is used to collect multimodal data of the objects to be detected, perform initial data division according to the classification rules, and form a multimodal classification sample data set of the objects to be detected; A feature fusion module, which is electrically connected to the data acquisition module, and is used to extract and splice image and text features of a multimodal classification sample data set of an object to be detected, and obtain a positive and negative sample matrix of an image and text feature vector of the object to be detected; A network search and matching module, the network search and matching module is electrically connected to the feature fusion module, and the network search and matching module is used to perform network item matching screening based on the image and text feature vector sample matrix of the item to be detected, and obtain a network reference item value sample set of the item to be detected; Model building module: The model building module is used to build a multimodal item value detection model based on the Transformer architecture; The value assessment module is electrically connected to the network search and matching module and the model building module. The value assessment module is used to train a multimodal item value detection model based on a network reference item value sample set of the item to be detected, substitute the graphic feature vector sample matrix of the item to be detected for fitting evaluation, and generate the value of the item to be detected.

5. According to claim 4, a multi-modal item value fitting evaluation system, It is characterized in that The feature fusion module includes: The classification unit extracts the image feature parameters and text feature parameters of the object to be detected based on the multimodal classification sample data set of the object to be detected according to the classification type, and marks a number index for each image feature parameter and text feature parameter; The feature concatenation unit aligns each image feature parameter and text feature parameter according to the marked number index, and concatenates the image feature parameters and text feature parameters with the same number index of the object to be detected into a graphic feature vector by an average weighted method, and combines them into a graphic feature vector sample set of the object to be detected; The positive and negative sample unit calculates the cosine similarity value of each pair of image-text feature vector samples in the image-text feature vector sample set of the object to be detected, and recursively traverses each pair of image-text feature vector samples to select the maximum similarity value of each pair of image-text feature vector samples as the positive sample of the image-text feature vector of the object to be detected, and selects the minimum similarity value of each pair of image-text feature vector samples as the negative sample of the image-text feature vector of the object to be detected; The positive and negative sample matrix unit combines the positive sample of the image-text feature vector of the object to be detected and the negative sample of the image-text feature vector of the object to be detected into a positive and negative sample matrix of the image-text feature vector of the object to be detected.

6. According to claim 4, a multi-modal item value fitting evaluation system, It is characterized in that The network search matching module includes: The transaction big data unit obtains historical transaction big data based on the API interface of the second-hand transaction platform; The transaction vector matrix unit, based on the historical transaction big data, performs linear mapping between the transaction items and transaction values ​​in the historical transaction big data according to the timestamp, and obtains the sample vector matrix of the network transaction item value; Referring to the sample set unit, each element in the positive and negative sample matrix of the image and text feature vector of the object to be detected is recursively matched with the network transaction item value sample vector matrix, and the network transaction item value sample vector corresponding to the maximum value of the positive and negative sample matching degree of the image and text feature vector of the detected object in the network transaction item value sample vector matrix is ​​screened out to form a network reference item value sample set for the object to be detected.

7. The multi-modal item value fitting evaluation system according to claim 4, It is characterized in that The value assessment module includes: A weighting unit, based on the network reference item value sample set of the item to be detected, assigns a weight to the feature of each network reference item value sample of the item to be detected by using an expectation maximization formula, and obtains a weighted feature set of the network reference item value sample of the item to be detected; The model training unit, based on the weighted feature set of the network reference item value samples of the items to be detected, substitutes the weighted features of the network reference item value samples of each item to be detected into the multimodal item value detection model, takes the actual value parameters of the network reference item value samples of each item to be detected as output, takes the minimum error function of the multimodal item value detection model as the training end target, and obtains the trained multimodal item value detection model; The item value assessment unit, based on the multimodal classification sample data set of the item to be detected, inputs the trained multimodal item value detection model to generate the assessment value parameters of the item to be detected.

Images