Comparison verification method and device for image-text data

Abstract

The invention discloses an image-text data comparison verification method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the following steps: acquiring first image-text data and second image-text data to be compared and verified; a first twin network is used for carrying out feature extraction and comparison on the first picture data and the second picture data, a picture comparison result is obtained, and the first twin network comprises a residual network module and a self-attention mechanism module; a second twin network is used for conducting feature extraction and comparison on the first text data and the second text data to obtain a text comparison result, and the second twin network is used for conducting attribute entity information extraction and alignment conversion; and obtaining a comparison verification result of the image-text data based on the image comparison result and the text comparison result. According to the embodiment, the multi-dimensional information of the image-text data is fully utilized to carry out image-text comparison verification, and the efficiency and accuracy of image-text comparison verification are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for comparing and verifying graphic and text data. Background Art

[0002] At present, in scenarios where image and text comparison and verification are required, when comparing and verifying image content and text content, the verification method is mostly to manually compare text attributes and image content; or configure some text attribute verification rules, and automatically compare some attributes based on the verification rules; or compare text and image content based on a multimodal large language model, etc.

[0003] However, these verification methods fail to fully utilize the multi-dimensional information of image and text data to perform image and text comparison verification, resulting in low efficiency and accuracy of image and text comparison verification. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method and device for comparing and verifying image and text data, which can perform feature extraction and comparison of image and text data by adopting a feature extraction network that integrates an optimized self-attention mechanism and a residual network, and combining attribute entity information extraction and a comparison model. It can make full use of the multi-dimensional information of the image and text data to perform image and text comparison and verification, thereby improving the efficiency and accuracy of image and text comparison and verification.

[0005] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for comparing and verifying graphic and text data is provided, comprising: Acquire first image and text data and second image and text data to be compared and verified, wherein the first image and text data includes first image data and first text data, and the second image and text data includes second image data and second text data; Using a first twin network to extract features and compare the first image data and the second image data to obtain an image comparison result, wherein the first twin network includes a residual network module and a self-attention mechanism module; Using a second twin network to perform feature extraction and comparison on the first text data and the second text data to obtain a text comparison result, wherein the second twin network is used for attribute entity information extraction and alignment conversion; A comparison and verification result of the image and text data is obtained based on the image comparison result and the text comparison result.

[0006] Optionally, the first twin network is used to perform feature extraction and comparison on the first image data and the second image data, including: using the residual network module in the first twin network to perform feature extraction on the first image data and the second image data, respectively, to obtain first image features and second image features; using the self-attention mechanism module in the first twin network to perform feature extraction on the first image data and the second image data, respectively, to obtain third image features and fourth image features; performing feature fusion on the first image features and the third image features to obtain first fused features, and performing feature fusion on the second image features and the fourth image features to obtain second fused features; and performing feature comparison based on the first fused features and the second fused features.

[0007] Optionally, feature fusion is performed on the first image feature and the third image feature to obtain a first fused feature, including: performing forward calculation on the first image feature and the third image feature to obtain a first weighted image feature and a third weighted image feature for subsequent weighting; when the dimensions of the first weighted image feature and the third weighted image feature are inconsistent, feature dimension alignment is performed on the first weighted image feature and the third weighted image feature; weighted summation is performed on the aligned first weighted image feature and the third weighted image feature to perform feature fusion to obtain a first fused feature; feature fusion is performed on the second image feature and the fourth image feature to obtain a second fused feature, including: performing forward calculation on the second image feature and the fourth image feature to obtain a second weighted image feature and a fourth weighted image feature for subsequent weighting; when the dimensions of the second weighted image feature and the fourth weighted image feature are inconsistent, feature dimension alignment is performed on the second weighted image feature and the fourth weighted image feature; weighted summation is performed on the aligned second weighted image feature and the fourth weighted image feature to perform feature fusion to obtain a second fused feature.

[0008] Optionally, the loss function of the first twin network is a triplet loss function, and a boundary parameter is introduced therein, and the loss function is obtained by dynamically adjusting the boundary parameter and sample weight.

[0009] Optionally, a second twin network is used to perform feature extraction and comparison on the first text data and the second data itself, including: using the second twin network to extract attribute entities on the first text data and the second data itself to obtain a first text attribute entity and a second text attribute entity; based on an attribute alias library, aligning and transforming the first text attribute entity and the second text attribute entity to obtain first text attribute features and second text attribute features of the same dimension; and performing feature comparison based on the first text attribute features and the second text attribute features.

[0010] Optionally, obtaining a comparison and verification result of the image and text data based on the image comparison result and the text comparison result includes: merging the image comparison result and the text comparison result to obtain a comparison and verification result of the image and text data.

[0011] According to another aspect of an embodiment of the present invention, there is provided a device for comparing and verifying graphic and text data, comprising: A data acquisition module, used to acquire first image and text data and second image and text data to be compared and verified, wherein the first image and text data includes first image data and first text data, and the second image and text data includes second image data and second text data; An image comparison module, used to use a first twin network to extract features and compare the first image data and the second image data to obtain an image comparison result, wherein the first twin network includes a residual network module and a self-attention mechanism module; A text comparison module, used to use a second twin network to perform feature extraction and comparison on the first text data and the second data to obtain a text comparison result, wherein the second twin network is used to perform attribute entity information extraction and alignment conversion; A result summary module is used to obtain a comparison and verification result of the image and text data based on the image comparison result and the text comparison result.

[0012] According to another aspect of an embodiment of the present invention, there is provided an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method for comparing and verifying graphic data provided by an embodiment of the present invention.

[0013] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for comparing and verifying graphic and text data provided by an embodiment of the present invention is implemented.

[0014] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method for comparing and verifying graphic and text data provided by an embodiment of the present invention is implemented.

[0015] An embodiment of the above invention has the following advantages or beneficial effects: by obtaining the first image and text data and the second image and text data to be compared and verified, the first image and text data include the first image data and the first text data, and the second image and text data include the second image data and the second text data; using the first twin network to extract and compare the first image data and the second image data to obtain the image comparison result, the first twin network includes a residual network module and a self-attention mechanism module; using the second twin network to extract and compare the first text data and the second data to obtain the text comparison result, the second twin network is used to extract and align the attribute entity information; a technical solution for obtaining the comparison and verification results of the image and text data based on the image comparison results and the text comparison results, by adopting a feature extraction network that integrates an optimized self-attention mechanism and a residual network, and combining the attribute entity information extraction and comparison model to perform feature extraction and comparison of the image and text data, the multi-dimensional information of the image and text data can be fully utilized to perform image and text comparison and verification, thereby improving the efficiency and accuracy of the image and text comparison and verification.

[0016] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention. Figure 1 Schematic diagram of main steps of a method for comparing and verifying graphic and text data according to an embodiment of the present invention; Figure 2 It is a schematic diagram of a picture data comparison and verification process according to an embodiment of the present invention; Figure 3 is a schematic diagram of a residual network module according to an embodiment of the present invention; Figure 4 It is a schematic diagram of a text data comparison and verification process according to an embodiment of the present invention; Figure 5 It is a schematic diagram of a comparison and verification process of graphic and text data according to an embodiment of the present invention; Figure 6 2 is a schematic diagram of main modules of a device for comparing and verifying graphic and text data according to an embodiment of the present invention; Figure 7 is an exemplary system architecture diagram to which embodiments of the present invention may be applied; Figure 8 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0019] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution disclosed in the present invention are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0020] Scenarios where image and text comparison and verification are required, such as the comparison of images and description information when products are put on shelves in the e-commerce field, the comparison of labels on logistics packages with cargo lists, the comparison of scanned paper medical records with electronic medical records, and so on. Taking the example of sending products before selling in retail scenarios, in the sales scenarios of some peripheral accessories of products, or in the sales scenarios of some maintenance products of products, these products are also needed to support the shipment of products, and they need to be bound and verified with the corresponding standard products of the products. The purpose of the verification is to ensure that the product can be accurately linked to the standard product, and whether the attributes, titles, pictures and other contents of the product are consistent with the corresponding attributes, titles, pictures maintained by the standard product or compatible and compliant, to ensure that the product is accurately bound to the standard product.

[0021] Currently, when verifying the binding between products and standard products, the verification method is mostly to manually compare the product text attributes and image content; or to configure some text attribute verification rules and automatically compare some attributes based on the verification rules; or to compare text and image content based on a multimodal large language model, etc.

[0022] However, these verification methods fail to fully utilize the multi-dimensional information of commodities and standard products to verify the delivery of products, resulting in low efficiency and accuracy of the verification of delivery of products. This is mainly reflected in the following four aspects:

[0023] 1. Limited manual verification capabilities: Manual verification relies heavily on the professional capabilities of maintenance personnel. An operation and maintenance personnel may be familiar with the verification of certain categories, but it is difficult for them to have a clear verification logic for all categories. At the same time, when new employees conduct verification, they are prone to misjudgment due to lack of experience. Currently, more and more product delivery scenarios are required, ranging from engine oil, engine filters, light bulbs to car covers, key chains and bags, etc., which increases the complexity of verification.

[0024] 2. Underutilization of image information: The images of SKUs (sold products with unique identifiers) of different merchants are very diverse, and the image information of SPUs (standardized product unique identifiers) focuses more on the product itself. Therefore, it is necessary to extract the features of SKU images and SPU images for comparison and verification. However, the current verification capability is limited to manual experience.

[0025] 3. Insufficient use of mutual verification of text information such as titles and attributes: There are similarities and differences between the SKU names, attributes and SPU names and attributes of the products sent by merchants. It is necessary to obtain accurate information that can represent the current product from the redundant SKU information and compare and verify it with the more standardized SPU names and attributes. The current text verification only uses part of the standardized attribute information and fails to make full use of the redundant information in the title text;

[0026] 4. No automated verification: The process cannot be automated, and the timeliness of merchant product verification cannot be guaranteed.

[0027] In order to solve the technical problems existing in the prior art, the present invention provides a method for comparing and verifying image and text data, which can make full use of the multi-dimensional image information and text information (for example, product name, product attribute information, etc.) of both sides of the image and text comparison to form similar products and different product pairs, and use a twin network optimized for this scenario to extract image features and text features, and perform compliance verification, thereby improving the efficiency and accuracy of image and text data comparison and verification; it solves the problems of inaccurate product delivery such as merchant product delivery errors, product mutual exclusion, attribute errors, product adaptation errors, etc. in the retail scenario product delivery process, improves product delivery standards and efficiency, reduces the adaptation error rate and order return volume caused by users purchasing inaccurate products, and improves customer satisfaction.

[0028] Figure 1 Schematic diagram of the main steps of the method for comparing and verifying graphic and text data according to an embodiment of the present invention. Figure 1 As shown, the comparison and verification method of graphic data in the embodiment of the present invention mainly includes the following steps S101 to S104.

[0029] Step S101: obtaining first image and text data to be compared and verified and second image and text data, wherein the first image and text data includes first image data and first text data, and the second image and text data includes second image data and second text data. In an embodiment of the present invention, when comparing and verifying image and text data, the first image and text data to be compared and verified can be obtained.

[0030] In one embodiment of the present invention, taking the delivery scenario of automobile maintenance and servicing products (also referred to as automobile products) as an example, it is necessary to determine the first image and text data corresponding to the automobile maintenance and servicing product SKU to be delivered and the second image and text data corresponding to the automobile standard product SPU to be bound, whether the image data in the two correspond to the same product, and whether the text description information such as the product description and product attribute description in the two correspond to the same product, so as to ensure the accuracy of the subsequent sales process and the product affiliation with the standard product.

[0031] Step S102: Use the first twin network to extract and compare features of the first image data and the second image data to obtain an image comparison result, wherein the first twin network includes a residual network module and a self-attention mechanism module. In an embodiment of the present invention, the first twin network is an improved twin network.

[0032] In the embodiments of the present invention, the use of traditional classification discrimination methods can solve the problem of whether images are of the same category to a certain extent. When the model is trained, the number of all categories will be known in advance, and the number is limited; however, for scenarios where there are many categories involved and the number of categories will continue to increase, traditional classification models are not applicable. For example, for the delivery verification of automobile products, there are many categories involved, and the number of categories involved is constantly increasing. If the classification model is used directly for processing, it is necessary to constantly re-fine-tune the model when adding new categories, and it is necessary to ensure that the accuracy of the original categories will not be affected while the accuracy of the new categories reaches the accuracy of use.

[0033] Therefore, the present invention uses an improved twin network to complete the comparison and verification of image data in such cases with a large number of unknown categories, which can fully consider the shooting angles, shooting lighting conditions, different distance sizes, and foreground and background of the two. At the same time, the backbone network for feature extraction of image data optimizes the model structure and loss function of the twin network under the automobile category to ensure the feasibility of application in the automobile category. Since the categories of automobile standard products are diverse and the shooting is not standardized, the present invention extracts attention features through the attention mechanism, and combines the multi-scale features extracted by the image feature extraction module for feature fusion, thereby increasing the feature extraction capability of the network.

[0034] According to one embodiment of the present invention, using the first twin network to perform feature extraction and comparison on the first image data and the second image data, specifically may include: using the residual network module in the first twin network to perform feature extraction on the first image data and the second image data, respectively, to obtain first image features and second image features; using the self-attention mechanism module in the first twin network to perform feature extraction on the first image data and the second image data, respectively, to obtain third image features and fourth image features; performing feature fusion on the first image features and the third image features to obtain first fused features, and performing feature fusion on the second image features and the fourth image features to obtain second fused features; performing feature comparison based on the first fused features and the second fused features.

[0035] Figure 2 FIG. 1 is a schematic diagram of a picture data comparison and verification process according to an embodiment of the present invention. Figure 2 As shown, in one embodiment of the present invention, image data comparison and verification is performed based on the improved twin network. Taking the comparison and verification of the image data P1 of the product SKU and the image data P2 of the standard product SPU as an example, after obtaining the image data P1 of the product SKU and the image data P2 of the standard product SPU, the image preprocessing will be performed first, such as image color enhancement, image cropping, etc. After the image preprocessing, the two branches of the twin network are used to extract features from the image data P1 of the product SKU and the image data P2 of the standard product SPU based on the feature extraction algorithm, and the image data P1 of the product SKU and the image data P2 of the standard product SPU are extracted based on the self-attention mechanism. Among them, the weights of the feature extraction models used by the feature extraction algorithms of the two branches of the twin network are the same, for example, W_p1; the entity extraction weights of the two branches of the twin network when performing feature extraction based on the self-attention mechanism are also the same, for example, W_p2.

[0036] After that, the two parts of the features extracted from the image data P1 of the product SKU are unified in size, and then the features are fused through a weighted algorithm to obtain the first fused feature; the two parts of the features extracted from the image data P2 of the standard product SPU are unified in size, and then the features are fused through a weighted algorithm to obtain the second fused feature. Finally, based on the first fused feature and the second fused feature, feature comparison is performed, and combined with the optimized ternary loss function, the verification results of the product image and the standard product image can be obtained.

[0037] In the present invention, the feature extraction model is used as the backbone network of the twin network, and as the feature extractors of the two branches respectively. For the convenience of explanation, the residual network is used to explain the feature extraction model. In the actual application process, this part can be replaced and iterated with other feature extraction models. For example, network models such as VGGNet, ResNet, MobileNet, EfficientNet, DenseNet, etc. can be used for feature extraction, but not limited to them.

[0038] The deep residual structure of the feature extraction model (taking the residual network as an example, different feature extraction models have their own special structures) can be used to capture the abstract features of the image. The most prominent feature of image processing is the residual module, as shown in the schematic diagram Figure 3 As shown, Figure 3 It is a schematic diagram of the residual network module of an embodiment of the present invention. Assuming that the input feature map is X, the feature map obtained after the residual block is F. The calculation process of the residual block first performs a convolution operation on the input feature map X to obtain an intermediate feature map M: M=Conv(X), then performs an activation function processing on the intermediate feature map M, such as the ReLU activation function: M'=ReLU(M), then, performs a convolution operation on the intermediate feature map [M'] to obtain the final feature map F: F=Conv(M'), and finally, adds the input feature map X to the obtained feature map F to obtain the output feature map Y of the residual block: Y=X+F. Through the processing of the residual block, the feature extraction network can better train the deep network, solve the gradient disappearance problem, and help improve the abstract feature extraction ability of product SKU and standard product SPU images.

[0039] Based on the feature extraction model, a self-attention mechanism is introduced for each branch of the twin network. There are many different objects and areas in the product image. Only the main body of the product image is more important for representing the content of the image. The self-attention mechanism can focus on the more important main part of the image and ignore the less important parts of the image, so that it can focus more on key information, thereby generating more accurate feature representations. The attention mechanism is used to dynamically learn the dependencies between input features and generate weighted feature representations.

[0040] The self-attention mechanism is used to dynamically learn the dependencies between the features of the input image sequence by calculating the attention weights of each position (such as each pixel in the image or each unit in the feature map) and other positions. The image pixel is used as a given input sequence or feature representation. ,in Indicates The feature representation of each position (i.e., pixel point), attention weight It can be obtained by the following calculation: in, It is a measure and The function of the degree of association between them is a function calculated by the neural network, by calculating the attention weight , we can get the weighted feature representation : Through the self-attention mechanism, the model can dynamically learn the dependencies between input features and generate weighted feature representations. , thereby capturing important information in the input product SKU and standard product SPU image data.

[0041] According to one embodiment of the present invention, feature fusion of the first image feature and the third image feature to obtain a first fused feature may specifically include: forward calculation of the first image feature and the third image feature to obtain a first weighted image feature and a third weighted image feature for subsequent weighting; when the dimensions of the first weighted image feature and the third weighted image feature are inconsistent, aligning the feature dimensions of the first weighted image feature and the third weighted image feature; and weighted summing the aligned first weighted image feature and the third weighted image feature to perform feature fusion to obtain the first fused feature.

[0042] The second image feature and the fourth image feature are subjected to feature fusion to obtain a second fused feature, which may specifically include: performing forward calculation on the second image feature and the fourth image feature to obtain a second weighted image feature and a fourth weighted image feature for subsequent weighting; when the dimensions of the second weighted image feature and the fourth weighted image feature are inconsistent, aligning the feature dimensions of the second weighted image feature and the fourth weighted image feature; and performing weighted summation on the aligned second weighted image feature and the fourth weighted image feature to perform feature fusion to obtain a second fused feature.

[0043] The residual network module and the self-attention mechanism of the feature extraction network are weightedly fused to complete the merging of abstract features with self-attention features. Here, a weighted strategy is used for fusion. At this time, for the two branches of the feature extraction network, there are two input images, namely and , through the features of the residual network branch and features branched by self-attention mechanism , after forward calculation, the features used for subsequent weighting are obtained and , there may be and In the case of inconsistent dimensions, it is necessary to further align the feature dimensions of the two. Next, a weighted strategy is used to fuse the two weighted features and perform a weighted sum of the two weighted features, namely: in, is the fusion weight, which can be a learned parameter or a fixed value set according to the specific task, used to complete the merging of abstract features and self-attention features. =0.5.

[0044] According to one embodiment of the present invention, the loss function of the first twin network is a triplet loss function, and a boundary parameter is introduced therein, and the loss function is obtained by dynamically adjusting the boundary parameter and sample weights.

[0045] In an embodiment of the present invention, for a twin network using a residual network and a self-attention mechanism as a feature extraction network, model training is performed by optimizing the original triplet loss function, wherein a "triplet" consists of an anchor sample (Anchor), a positive sample (Positive) and a negative sample (Negative). The feature vectors of the anchor sample A, the positive sample N, and the negative sample P are respectively represented as feature vectors f(A), f(P) and f(N), and the optimization is performed by dynamically adjusting the boundary parameter margin and considering the sample weight.

[0046] In the triple loss function, a dynamic boundary parameter margin is introduced so that the margin can be adjusted dynamically as the training progresses. Let the margin used be β, and set β to a dynamic variable, for example, gradually decrease with the increase of the number of training rounds. In this way, the model can pay more attention to the relative distance between samples in the early stage of training, and gradually relax the margin as the training progresses, so that the model pays more attention to the relative order between samples.

[0047] At the same time, when calculating the triplet loss, different weights are given to different samples. Suppose I have a weight function , where x represents the relationship between samples, and the weight can be adjusted dynamically according to the similarity or difficulty between samples. The similarity between samples is used as the weight, the higher the similarity, the greater the weight of the sample, and the lower the similarity, the smaller the weight of the sample.

[0048] Taking into account the dynamic adjustment of margin and the optimization of sample weights, the formula of the triplet loss function can be expressed as: in, represents the sample weight function, express , express The square of the L2 norm of . By dynamically adjusting and sample weights, can better utilize the features extracted by the optimized feature extraction network and improve the performance of the model.

[0049] The feature extraction network structure change, feature fusion and loss function optimization are completed, that is, the image verification model part of the improved twin network is optimized, and the historical verification data is used to train the model until convergence, which is used to determine the similarity of the same standard product SPU of the product SKU, and set the similarity threshold δ. If the threshold is exceeded, the image comparison verification passes, otherwise the verification fails.

[0050] Step S103: Use the second twin network to extract and compare features of the first text data and the second text data to obtain text comparison results. The second twin network is used to extract attribute entity information and align conversion.

[0051] According to one embodiment of the present invention, the first text data and the second data itself are feature extracted and compared using a second twin network, which may specifically include: using the second twin network to extract attribute entities from the first text data and the second data itself to obtain a first text attribute entity and a second text attribute entity; based on an attribute alias library, aligning and transforming the first text attribute entity and the second text attribute entity to obtain first text attribute features and second text attribute features of the same dimension; and performing feature comparison based on the first text attribute features and the second text attribute features.

[0052] Figure 4 1 is a schematic diagram of a text data comparison and verification process according to an embodiment of the present invention. Figure 4 As shown, in an embodiment of the present invention, the comparison and verification of the text data T1 of the product SKU and the text data T2 of the standard product SPU is taken as an example. First, the text data T1 of the product SKU and the text data T2 of the standard product SPU are obtained, and then the text preprocessing is performed, for example, including invalid text deletion, blank value filling, and so on. After the text preprocessing, the text data T1 of the product SKU and the text data T2 of the standard product SPU are respectively extracted for verification based on the two branches of the twin network. Among them, the weights of the feature extraction models used by the two branches of the twin network for attribute entity extraction are the same, for example, W_t.

[0053] Afterwards, based on the attribute alias mapping, the attribute entity corresponding to the text data T1 of the product SKU is standardized and converted to obtain the attribute entity Tn1, and the attribute entity corresponding to the text data T2 of the standard product SPU is standardized and converted to obtain the attribute entity Tn2. Then, based on the standardized attribute entity comparison of attribute entity Tn1 and attribute entity Tn2, the attribute verification results of the product text and the standard product text are obtained.

[0054] In one embodiment of the present invention, when a product SKU is shipped, in addition to ensuring the accuracy of the image, it is also necessary to verify the accuracy of the attributes in the text. Therefore, named entity extraction plus alias library maintenance is used to extract product and standard product attribute information in the text to verify the information of the text dimension.

[0055] The text T1 for the product SKU can be expressed as , whose length is , and the text T2 of the standard product SPU can be expressed as , whose length is , use BIO annotation to extract the attribute entities involved in the commodities and standard product categories. BIO annotation is a commonly used natural language processing annotation method used to represent the part of speech and morphological information of a word in a sentence.

[0056] In an embodiment of the present invention, an entity recognition model can be pre-trained to perform attribute entity extraction on the text data T1 of the commodity SKU and the text data T2 of the standard product SPU. Among them, when training the entity recognition model, the training data is first prepared, and the text sequence needs to be annotated using the BIO annotation method. The BIO annotation method is a commonly used named entity annotation method, in which B represents the beginning of the entity, I represents the middle part of the entity, and O represents the non-entity part. For each text sequence, it is necessary to mark out the named entities therein. In the entity extraction of commodities and standard products, it is necessary to complete the annotation of important comparison information entities such as brand, specification model, body type, style, etc.

[0057] After that, feature extraction is performed on the training data. The pre-trained BERT (Bidirectional Encoder Representations from Transformers, language representation model) model can be used to extract features from text sequences. For each text sequence, it is converted into the input format of the BERT model, that is, special [CLS] and [SEP] tags are added, and segmentation and position embedding are performed, and then input into the BERT model to obtain context-related word vector representations. Among them, special tags are used to distinguish the input position and the end position of the sentence. The [CLS] tag is a special tag used to indicate the beginning of the entire sequence. In BERT's pre-training and many downstream tasks (such as classification tasks), the output vector of the [CLS] tag is usually used to represent the aggregate information of the entire input sequence. It is always placed at the front of the input sequence. The [SEP] tag is used to separate two sentences. In BERT's pre-training tasks and some downstream tasks (such as sentence pair classification), the [SEP] tag is used to distinguish two sentences or sentence fragments. It is placed at the end of each sentence or between two sentences.

[0058] Then, for each labeled position, the output of the BERT model is used as input, and a sequence labeling model (such as a CRF layer) is used to predict the label of each word to complete the classification task for each word. In this way, an entity recognition model can be trained based on the sequence labeling model.

[0059] For new text sequences T1 and T2, the trained entity recognition model is used for classification prediction to obtain the recognition results of named entities in the text sequences. These results will be annotated in BIO annotation mode, indicating whether each word belongs to a named entity and its position in the named entity.

[0060] Through the above steps, BERT+CRF can be used to extract the attribute entities of the text T1 of the product SKU and the text T2 of the standard product SPU, which are represented as N T1 = [ n 11 , n 12 , … , n 1 i ] and N T2 = [ n 21 , n 22 , … , n 2 j ] At the same time, through the attribute alias library pre-maintained in the database, and The existing entities in the standardization conversion are converted into attribute features of the same dimension for subsequent comparison. For a specific category, the attributes that need to be compared have a fixed number. , through the comparison function Equal, Equal is used to verify and Whether the corresponding attribute characteristics are consistent, return the number of attributes that are consistent with the verification ,if , then the text verification of the product SKU's text T1 and the standard product SPU's text T2 passes, otherwise the verification fails.

[0061] Step S104: obtaining a comparison and verification result of the image and text data based on the image comparison result and the text comparison result.

[0062] According to an embodiment of the present invention, obtaining a comparison and verification result of the image and text data based on the image comparison result and the text comparison result may specifically include: merging the image comparison result and the text comparison result to obtain a comparison and verification result of the image and text data. Specifically, if both the image comparison result and the text comparison result are comparison passed, the comparison and verification result of the image and text data is comparison and verification passed; otherwise, the comparison and verification result of the image and text data is comparison and verification failed.

[0063] The above improved twin network's image verification module and attribute entity extraction text verification module have completed the verification results of the image data and text data to be compared. Both need to be verified to pass in order to ensure that the image and text data verification passes. At the same time, for image and text data that failed the verification, the reason for the comparison verification failure needs to be specified, such as: image verification error, brand verification error, vehicle attribute adaptation verification and other specific attribute verification failure reasons, which are used to guide merchants who failed the verification to modify the product information to improve the accuracy of product delivery.

[0064] Figure 5 The figure is a schematic diagram of the comparison and verification process of the graphic data of an embodiment of the present invention. In the embodiment of the present invention, in order to solve the problem of automating the verification process while ensuring the accuracy and efficiency of the verification during the verification of automobile products, the comparison and verification process of the graphic data provided by the present invention mainly includes the following steps: Step 1: The system receives the product SKU and standard product SPU to be matched; Step 2: The system loads the image data and text data such as title, specification attributes and extended attribute content of the product SKU from the storage data source of the product and standard product, as well as the image data and text data of the same dimensions of the standard product SPU; Step 3: For the image data P1 of the product SKU and the image data P2 of the standard product SPU, after image preprocessing, use the twin network improved based on the residual network and self-attention mechanism to extract image features, and perform image feature comparison based on the image feature measurement function to obtain the image comparison verification results of the product image and the standard product image; Step 4: For the text data T1 of the product SKU and the text data T2 of the standard product SPU, after text preprocessing, use the twin network structure to extract attribute entities, and then, based on the attribute alias library, use the text entity measurement function to perform text feature comparison to obtain the text comparison verification results of the product text and the standard product text; Step 5: Based on the image comparison and verification results of the product image and the standard product image and the text comparison and verification results of the product text and the standard product text, the image comparison and verification results and the text comparison and verification results are merged to obtain the comparison and verification results of the image and text data; Step 6: Output the comparison and verification results of the graphic and text data. If the comparison and verification result is that the verification is passed, a prompt message of the verification passing is output. If the comparison and verification result is that the verification fails, the reason for the verification failure is also output, so that the graphic and text information of the product can be modified before sending the product.

[0065] Figure 6 Schematic diagram of the main modules of the device for comparing and verifying graphic and text data according to an embodiment of the present invention. Figure 6 As shown, the device 600 for comparing and verifying graphic and text data in the embodiment of the present invention mainly includes a data acquisition module 601 , a picture comparison module 602 , a text comparison module 603 and a result summary module 604 .

[0066] The data acquisition module 601 is used to acquire first image and text data and second image and text data to be compared and verified, wherein the first image and text data includes first image data and first text data, and the second image and text data includes second image data and second text data; An image comparison module 602 is used to extract features and compare the first image data and the second image data using a first twin network to obtain an image comparison result, wherein the first twin network includes a residual network module and a self-attention mechanism module; A text comparison module 603 is used to extract and compare the first text data and the second text data using a second twin network to obtain a text comparison result, wherein the second twin network is used to extract attribute entity information and perform alignment conversion; The result summary module 604 is used to obtain the comparison and verification result of the image and text data based on the image comparison result and the text comparison result.

[0067] According to one embodiment of the present invention, the image comparison module 602 can be specifically used to: use the residual network module in the first twin network to perform feature extraction on the first image data and the second image data respectively, to obtain first image features and second image features; use the self-attention mechanism module in the first twin network to perform feature extraction on the first image data and the second image data respectively, to obtain third image features and fourth image features; perform feature fusion on the first image features and the third image features to obtain first fusion features, and perform feature fusion on the second image features and the fourth image features to obtain second fusion features; perform feature comparison based on the first fusion features and the second fusion features.

[0068] According to another embodiment of the present invention, the image comparison module 602 includes a feature fusion unit (not shown in the figure), which is used to: perform forward calculation on the first image feature and the third image feature to obtain the first weighted image feature and the third weighted image feature for subsequent weighting; when the dimensions of the first weighted image feature and the third weighted image feature are inconsistent, align the feature dimensions of the first weighted image feature and the third weighted image feature; perform weighted summation on the aligned first weighted image feature and the third weighted image feature to perform feature fusion to obtain the first fused feature. In addition, the feature fusion unit (not shown in the figure) is also used to: perform forward calculation on the second image feature and the fourth image feature to obtain the second weighted image feature and the fourth weighted image feature for subsequent weighting; when the dimensions of the second weighted image feature and the fourth weighted image feature are inconsistent, align the feature dimensions of the second weighted image feature and the fourth weighted image feature; perform weighted summation on the aligned second weighted image feature and the fourth weighted image feature to perform feature fusion to obtain the second fused feature.

[0069] According to another embodiment of the present invention, the loss function of the first twin network in the figure is a triple loss function, and a boundary parameter is introduced therein. The loss function is obtained by dynamically adjusting the boundary parameter and sample weight.

[0070] According to another embodiment of the present invention, the text comparison module 603 can be specifically used to: use a second twin network to extract attribute entities from the first text data and the second text data to obtain a first text attribute entity and a second text attribute entity; based on an attribute alias library, align and transform the first text attribute entity and the second text attribute entity to obtain a first text attribute feature and a second text attribute feature of the same dimension; perform feature comparison based on the first text attribute feature and the second text attribute feature.

[0071] According to another embodiment of the present invention, the result aggregation module 604 may also be used to: merge the image comparison result and the text comparison result to obtain a comparison verification result of the image and text data.

[0072] According to the technical solution of an embodiment of the present invention, by obtaining the first image and text data and the second image and text data to be compared and verified, the first image and text data include the first image data and the first text data, and the second image and text data include the second image data and the second text data; using the first twin network to extract and compare the first image data and the second image data to obtain the image comparison result, the first twin network includes a residual network module and a self-attention mechanism module; using the second twin network to extract and compare the first text data and the second data to obtain the text comparison result, the second twin network is used to extract and align the attribute entity information; the technical solution of obtaining the comparison and verification result of the image and text data based on the image comparison result and the text comparison result, by adopting the feature extraction network that integrates the optimized self-attention mechanism and the residual network, and combining the attribute entity information extraction and comparison model to perform feature extraction and comparison of the image and text data, the multi-dimensional information of the image and text data can be fully utilized to perform image and text comparison and verification, thereby improving the efficiency and accuracy of the image and text comparison and verification.

[0073] Figure 7 An exemplary system architecture 700 is shown to which the method for comparing and verifying image and text data or the device for comparing and verifying image and text data according to the embodiment of the present invention can be applied.

[0074] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, 703, network 704 and server 705. Network 704 is used to provide a medium for communication links between terminal devices 701, 702, 703 and server 705. Network 704 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0075] The user can use the terminal devices 701, 702, 703 to interact with the server 705 through the network 704 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 701, 702, 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0076] The terminal devices 701 , 702 , and 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0077] Server 705 can be a server that provides various services, such as a backend management server that provides support for shopping websites browsed by users using terminal devices 701, 702, and 703 (for example only). The backend management server can obtain the first image and text data and the second image and text data to be compared and verified for the received comparison and verification request for image and text data, wherein the first image and text data includes the first image data and the first text data, and the second image and text data includes the second image data and the second text data; use the first twin network to extract features and compare the first image data and the second image data to obtain an image comparison result, wherein the first twin network includes a residual network module and a self-attention mechanism module; use the second twin network to extract features and compare the first text data and the second data to obtain a text comparison result, wherein the second twin network is used to extract attribute entity information and align and convert; obtain the comparison and verification result of the image and text data based on the image comparison result and the text comparison result, and feed back the processing result (such as the comparison and verification result of the image and text data - for example only) to the terminal device.

[0078] It should be noted that the comparison and verification method of the graphic data provided in the embodiment of the present invention is generally executed by the server 705 , and accordingly, the comparison and verification device of the graphic data is generally set in the server 705 .

[0079] It should be understood that Figure 7 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0080] Reference below Figure 8 , which shows a schematic diagram of the structure of a computer system 800 of a terminal device or server suitable for implementing an embodiment of the present invention. Figure 8 The terminal device or server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0081] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0082] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0083] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-mentioned functions defined in the system of the present invention are executed.

[0084] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0085] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0086] The units or modules involved in the embodiments of the present invention may be implemented in software or hardware. The units or modules described may also be provided in a processor, for example, may be described as: a processor including a data acquisition module, an image comparison module, a text comparison module and a result summary module. The names of these units or modules do not, in certain cases, constitute limitations on the units or modules themselves, for example, the result summary module may also be described as "a module for obtaining comparison and verification results of graphic and text data based on the image comparison results and the text comparison results".

[0087] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device includes: obtaining the first image data and the second image data to be compared and verified, the first image data includes the first image data and the first text data, and the second image data includes the second image data and the second text data; using the first twin network to extract features and compare the first image data and the second image data to obtain the image comparison result, the first twin network includes a residual network module and a self-attention mechanism module; using the second twin network to extract features and compare the first text data and the second data to obtain the text comparison result, the second twin network is used to extract attribute entity information and align the conversion; based on the image comparison result and the text comparison result, the comparison and verification result of the image data is obtained.

[0088] According to the technical solution of an embodiment of the present invention, by obtaining the first image and text data and the second image and text data to be compared and verified, the first image and text data include the first image data and the first text data, and the second image and text data include the second image data and the second text data; using the first twin network to extract and compare the first image data and the second image data to obtain the image comparison result, the first twin network includes a residual network module and a self-attention mechanism module; using the second twin network to extract and compare the first text data and the second data to obtain the text comparison result, the second twin network is used to extract and align the attribute entity information; the technical solution of obtaining the comparison and verification result of the image and text data based on the image comparison result and the text comparison result, by adopting the feature extraction network that integrates the optimized self-attention mechanism and the residual network, and combining the attribute entity information extraction and comparison model to perform feature extraction and comparison of the image and text data, the multi-dimensional information of the image and text data can be fully utilized to perform image and text comparison and verification, thereby improving the efficiency and accuracy of the image and text comparison and verification.

[0089] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for comparing and verifying graphic data, characterized in that: include: Acquire first image and text data and second image and text data to be compared and verified, wherein the first image and text data includes first image data and first text data, and the second image and text data includes second image data and second text data; Using a first twin network to extract features and compare the first image data and the second image data to obtain an image comparison result, wherein the first twin network includes a residual network module and a self-attention mechanism module; Using a second twin network to perform feature extraction and comparison on the first text data and the second text data to obtain a text comparison result, wherein the second twin network is used for attribute entity information extraction and alignment conversion; A comparison and verification result of the image and text data is obtained based on the image comparison result and the text comparison result.

2. The method according to claim 1, characterized in that Using a first twin network to extract and compare features of the first image data and the second image data includes: Use the residual network module in the first twin network to perform feature extraction on the first image data and the second image data respectively to obtain first image features and second image features; Use the self-attention mechanism module in the first twin network to extract features from the first image data and the second image data respectively to obtain third image features and fourth image features; Performing feature fusion on the first picture feature and the third picture feature to obtain a first fused feature, and performing feature fusion on the second picture feature and the fourth picture feature to obtain a second fused feature; Feature comparison is performed based on the first fusion feature and the second fusion feature.

3. The method according to claim 2, characterized in that Performing feature fusion on the first picture feature and the third picture feature to obtain a first fused feature includes: Performing forward calculation on the first picture feature and the third picture feature to obtain a first weighted picture feature and a third weighted picture feature for subsequent weighting; When the dimensions of the first weighted picture feature and the third weighted picture feature are inconsistent, aligning the feature dimensions of the first weighted picture feature and the third weighted picture feature; Performing weighted summation on the aligned first weighted image feature and the third weighted image feature to perform feature fusion to obtain a first fusion feature; Performing feature fusion on the second picture feature and the fourth picture feature to obtain a second fused feature includes: Performing forward calculation on the second picture feature and the fourth picture feature to obtain a second weighted picture feature and a fourth weighted picture feature for subsequent weighting; When the dimensions of the second weighted picture feature and the fourth weighted picture feature are inconsistent, aligning the feature dimensions of the second weighted picture feature and the fourth weighted picture feature; A weighted sum is performed on the aligned second weighted image features and the fourth weighted image features to perform feature fusion to obtain a second fused feature.

4. The method according to any one of claims 1 to 3, characterized in that The loss function of the first twin network is a triplet loss function, in which a boundary parameter is introduced. The loss function is obtained by dynamically adjusting the boundary parameter and sample weight.

5. The method according to claim 1, characterized in that Using a second twin network to extract and compare features of the first text data and the second text data includes: Using a second twin network to extract attribute entities from the first text data and the second text data, to obtain a first text attribute entity and a second text attribute entity; Based on the attribute alias library, the first text attribute entity and the second text attribute entity are aligned and transformed to obtain a first text attribute feature and a second text attribute feature of the same dimension; A feature comparison is performed based on the first text attribute feature and the second text attribute feature.

6. The method according to claim 1, characterized in that The comparison and verification result of the image and text data is obtained based on the image comparison result and the text comparison result, including: The image comparison result and the text comparison result are combined to obtain a comparison verification result of the image and text data.

7. A device for comparing and verifying graphic and text data, characterized in that: include: A data acquisition module, used to acquire first image and text data and second image and text data to be compared and verified, wherein the first image and text data includes first image data and first text data, and the second image and text data includes second image data and second text data; An image comparison module, used to use a first twin network to extract features and compare the first image data and the second image data to obtain an image comparison result, wherein the first twin network includes a residual network module and a self-attention mechanism module; A text comparison module, used to use a second twin network to perform feature extraction and comparison on the first text data and the second data to obtain a text comparison result, wherein the second twin network is used to perform attribute entity information extraction and alignment conversion; A result summary module is used to obtain a comparison and verification result of the image and text data based on the image comparison result and the text comparison result.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images