Cargo picture retrieval method, device and equipment and storage medium

Through the multimodal regularization module, the feature fusion training of cargo pictures and text information is performed, and the marginal value of the loss function is dynamically adjusted, which solves the problem of low retrieval accuracy of existing cargo pictures, and achieves higher matching accuracy and retrieval accuracy .

CN119025697APending Publication Date: 2024-11-26SHENZHEN LEAPFROG NEW TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410783615.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-18
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The accuracy of existing cargo image retrieval methods is not high, especially when dealing with complex and diverse cargo images, which is mainly due to the neglect of the interaction between modals and the influence of clustering algorithm results.

Method used

The multimodal regularization module is adopted to train the features generated by the text encoder and the image encoder in fusion, and the marginal value of the loss function is dynamically adjusted to improve the matching accuracy using the triple loss function.

Benefits of technology

It improves the matching accuracy and retrieval accuracy of the cargo pictures and corresponding text information, provides more comprehensive and rich feature expression, and enhances the semantic understanding ability of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025697A_ABST
    Figure CN119025697A_ABST
Patent Text Reader

Abstract

The invention discloses a goods picture retrieval method, device and equipment and a storage medium. The goods picture retrieval method comprises the steps of obtaining a to-be-retrieved goods picture; inputting the to-be-retrieved goods picture into a goods picture retrieval model to obtain goods text data corresponding to the to-be-retrieved goods picture; the goods picture retrieval model is obtained by constructing a multi-modal regularization module, and performing multi-modal feature fusion training on text features generated by goods text sample data through a text encoder and image features generated by goods picture sample data through an image encoder; the loss function in the training comprises a triple loss function, and the marginal value of the triple loss function is dynamically adjusted according to the stage of the training and the improvement of the prediction performance. By using the goods picture retrieval method disclosed by the invention, the accuracy of goods picture retrieval can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of logistics technology, and in particular to a cargo image retrieval method, device, equipment and storage medium. Background Art

[0002] In the logistics industry, the flow of goods is mainly recorded and tracked by waybills. During the flow of goods, some goods may have waybills lost or the information on them cannot be identified due to various reasons, making it impossible to determine the recipient or sender of the goods, resulting in unidentified goods. This reduces the efficiency of goods delivery and provides a poor user experience.

[0003] With the advancement of artificial intelligence and image recognition technology, it is now possible to identify and retrieve cargo images to determine cargo-related information, so as to deal with the problem of missing items in the cargo circulation process. The identification and retrieval of cargo images refers to retrieving images matching specific cargo from a large number of images through the features of cargo images and the constructed feature library, so as to query the consignee and consignee information of the cargo in the logistics system. There are two main types of existing cargo image retrieval methods. One is the method that relies on image information and a single-modal encoder. However, this method ignores the interaction between modalities and has a low retrieval accuracy when dealing with complex and diverse cargo images. The other is the method based on multimodal regularization. The model performance of this method depends to a certain extent on the results of the clustering algorithm. The results of the clustering algorithm may be affected by sample distribution and noise, thereby affecting the model training effect and the retrieval accuracy is also low. Summary of the invention

[0004] The present invention provides a cargo image retrieval method, device, equipment and storage medium, which are used to solve the technical problem of low accuracy of cargo image retrieval in the prior art.

[0005] In order to solve the above technical problems, in a first aspect, the present invention provides a cargo image retrieval method, the cargo image retrieval method comprising:

[0006] Get the pictures of the goods to be retrieved;

[0007] The cargo picture to be retrieved is input into a cargo picture retrieval model to obtain cargo text data corresponding to the cargo picture to be retrieved; the cargo picture retrieval model is obtained by constructing a multimodal regularization module, and performing multimodal feature fusion training on text features generated by a text encoder for cargo text sample data, and on image features generated by an image encoder for cargo picture sample data, wherein the loss function in the training includes a triplet loss function, and the marginal value of the triplet loss function is dynamically adjusted according to the stage of training and the improvement of prediction performance.

[0008] Optionally, the loss function in the training includes a triplet loss function, and the marginal value of the triplet loss function is dynamically adjusted according to the stage of training and the improvement of prediction performance, including:

[0009] The formula of the loss function is:

[0010] Loss = l(x a ,x p ,x n ,α)+l BCE

[0011] Among them, l(x a ,x p ,x n ,α) is the triple loss function, x a is the anchor point data, x p is the positive sample data, x n is the negative sample data, α is the marginal value; l BCE is a binary cross entropy loss function, which calculates whether the text feature and the image feature match, and is used for training the multimodal regularization module;

[0012] The marginal value of the triplet loss function is adjusted according to the stage of training, the verification accuracy of the current time step, and the boost coefficient of different training stages and the marginal value dynamic adjustment formula determined by the preset boost threshold.

[0013] Optionally, the marginal value dynamic adjustment formula includes:

[0014]

[0015] Among them, α t+1 is the marginal value at time step t+1, α t is the marginal value at time step t, Δp t is the verification accuracy p of the current time step t+1 Compared with the verification accuracy p of the previous time step t , Δp1 is the preset performance improvement threshold in the early stage of training, Δp2 is the preset performance improvement threshold in the middle stage of training, Δp3 is the preset performance improvement threshold in the late stage of training, c1 is the improvement coefficient of the marginal value increase in the early stage of training, c2 is the improvement coefficient of the marginal value increase in the middle stage of training, c3 is the improvement coefficient of the marginal value increase in the late stage of training, and Δp1<Δp2<Δp3, c1>c2>c3.

[0016] Optionally, the cargo image retrieval method further includes:

[0017] The text encoder includes a hierarchical Transformer encoder and a first Transformer decoder constructed by combining a self-attention mechanism and a mutual attention mechanism;

[0018] The hierarchical Transformer encoder encodes the sender information, address information and cargo information in the cargo text sample data, and the encoded information is processed by the first Transformer decoder to generate the text features.

[0019] Optionally, the hierarchical Transformer encoder encodes the sender information, address information and cargo information in the cargo text sample data, and the encoded information is processed by the first Transformer decoder to generate the text features, including:

[0020] The first Transformer encoder in the hierarchical Transformer encoder respectively performs the first encoding on the sender information, the address information and the cargo information, and correspondingly outputs the first encoded sender information, the first encoded address information and the first encoded cargo information;

[0021] The second Transformer encoder in the hierarchical Transformer encoder respectively performs a second encoding on the address information after the first encoding and the cargo information after the first encoding, and correspondingly outputs the address information after the second encoding and the cargo information after the second encoding;

[0022] One of the sender information after the first encoding, the address information after the second encoding, and the cargo information after the second encoding is used as a query, and the remaining two are input into the first Transformer decoder for processing as a combination of Key and Value, and the processed outputs are spliced ​​to obtain the text feature.

[0023] Optionally, the cargo image retrieval method further includes:

[0024] The multimodal regularization module includes a second Transformer decoder, which combines the cross-attention mechanism to perform inter-modal information fusion and regularization processing on the text features and image features.

[0025] Optionally, the cargo image retrieval method further includes:

[0026] The visual Transformer is used as the image encoder to convert the cargo image sample data into a series of image Tokens to generate the image features.

[0027] In a second aspect, the present invention provides a cargo image retrieval device, comprising a data acquisition module and a retrieval module;

[0028] The data acquisition module acquires the image of the goods to be retrieved;

[0029] The retrieval module is used to input the cargo image to be retrieved into the cargo image retrieval model to obtain cargo text data corresponding to the cargo image to be retrieved; the cargo image retrieval model is obtained by constructing a multimodal regularization module, and performing multimodal feature fusion training on the text features generated by the cargo text sample data via a text encoder, and the image features generated by the cargo image sample data via an image encoder. The loss function in the training includes a triplet loss function, and the marginal value of the triplet loss function is dynamically adjusted according to the stage of training and the improvement of prediction performance.

[0030] In a third aspect, the present invention provides a cargo image retrieval device, including a memory and a processor, wherein:

[0031] The memory is used to store computer programs;

[0032] The processor is used to read the program in the memory and execute the steps of the cargo image retrieval method provided in the first aspect above.

[0033] In a fourth aspect, the present invention provides a computer-readable storage medium having a readable computer program stored thereon, which, when executed by a processor, implements the steps of the cargo image retrieval method provided in the first aspect above.

[0034] Compared with the prior art, the cargo image retrieval method, device, equipment and storage medium provided by the present invention have the following beneficial effects:

[0035] Obtain a picture of goods to be retrieved; input the picture of goods to be retrieved into a goods picture retrieval model to obtain goods text data corresponding to the picture of goods to be retrieved; the goods picture retrieval model is obtained by constructing a multimodal regularization module, and performing multimodal feature fusion training on text features generated by a text encoder for sample data of goods text, and on image features generated by an image encoder for sample data of goods pictures, wherein the loss function in the training includes a triplet loss function, and the marginal value of the triplet loss function is dynamically adjusted according to the stage of training and the improvement of prediction performance; the method provides a more comprehensive and rich feature expression by multimodal feature fusion of text features and image features through a multimodal regularization module, and at the same time, in combination with the training, the marginal value of the triplet loss function is dynamically adjusted according to the stage of training and the improvement of prediction performance, thereby improving the matching accuracy between the goods picture and the corresponding goods text, and improving the accuracy of goods picture retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only part of the embodiments of the present invention, rather than all of the embodiments. For ordinary technicians in this field, without paying creative work, other drawings obtained based on these drawings all fall within the scope of protection of this application.

[0037] Figure 1 It is a flow chart of a cargo image retrieval method provided by an embodiment of the present invention;

[0038] Figure 2 It is a structural schematic diagram of a cargo image retrieval model provided by an embodiment of the present invention;

[0039] Figure 3 is a processing schematic diagram of a text editor provided by an embodiment of the present invention;

[0040] Figure 4 It is a structural schematic diagram of a cargo image retrieval device provided by an embodiment of the present invention;

[0041] Figure 5 It is a structural schematic diagram of a cargo image retrieval device provided by an embodiment of the present invention;

[0042] Figure 6 It is a structural schematic diagram of a computer-readable storage medium provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0044] In order to make the description of the disclosed content more detailed and complete, the following is an illustrative description of the implementation mode and specific examples of the present invention; however, this is not the only form of implementing or applying the specific embodiments of the present invention. The implementation mode covers the features of multiple specific embodiments and the method steps and their sequence for constructing and operating these specific embodiments. However, other specific embodiments can also be used to achieve the same or equal functions and step sequences. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0045] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0046] In the description of the embodiments of the present invention, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two, and other quantifiers are similar. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention, and the embodiments of the present application and the features in the embodiments may be combined with each other without conflict.

[0047] Example 1

[0048] like Figure 1 As shown, it is a flow chart of a cargo image retrieval method provided by an embodiment of the present invention, and the cargo image retrieval method includes the following steps.

[0049] Step S101, obtaining pictures of goods to be retrieved;

[0050] For goods whose waybill information is lost or unrecognizable, first take a photo of the goods in the open cargo box to obtain the image of the goods to be retrieved. For example, when taking photos, the cargo box with the goods is placed with the opening facing upwards, and the camera is facing the cargo box opening. All the goods in the cargo box need to be photographed, and the cargo box needs to account for more than 70% of the picture.

[0051] Step S102, input the cargo picture to be retrieved into the cargo picture retrieval model to obtain cargo text data corresponding to the cargo picture to be retrieved; the cargo picture retrieval model is obtained by constructing a multimodal regularization module, and performing multimodal feature fusion training on the text features generated by the cargo text sample data via the text encoder, and on the image features generated by the cargo picture sample data via the image encoder. The loss function in the training includes a triplet loss function, and the marginal value of the triplet loss function is dynamically adjusted according to the stage of training and the improvement of prediction performance.

[0052] The cargo image to be retrieved is input into the trained cargo image retrieval model to obtain cargo text data corresponding to the cargo image to be retrieved. According to the cargo text data predicted by the cargo image retrieval model, the waybill information of the cargo can be determined. The cargo image retrieval model is trained using the collected cargo text sample data and cargo image sample data. Among them, the collected cargo text sample data includes the sender information, address information and cargo information of the cargo. For example, the sender information is the name of the sender unit or the name of the sender, the address information includes the mailing address information and the receiving address information, and the cargo information includes the cargo type information and / or the cargo weight information; the collected cargo image sample data is the cargo image obtained by taking a photo of the cargo in the open cargo box. The collection method is the same as the method of obtaining the cargo image to be retrieved. For example, when collecting the cargo image sample data for taking a photo, the cargo box opening for the cargo is placed upward, and the camera is facing the cargo box opening. All the cargo in the cargo box needs to be photographed, and the proportion of the cargo box in the picture needs to be greater than 70%.

[0053] The cargo image retrieval model is obtained by constructing a multimodal regularization module, generating text features of cargo text sample data via a text encoder, and generating image features of cargo image sample data via an image encoder, and performing multimodal feature fusion training on the model according to a loss function.

[0054] like Figure 2The figure shows a schematic diagram of the structure of the cargo image retrieval model, which includes a multimodal regularization module, an image editor, and a text editor. During the training of the cargo image retrieval model, the cargo text sample data is generated into text features via a text encoder, and the cargo image sample data is generated into image features via an image encoder. Then, the multimodal regularization module performs multimodal feature fusion on the text features and the image features to obtain fusion features. During the iterative process of the model performing multimodal fusion training on the fusion features, the marginal value of the triplet loss function in the loss function of the model is dynamically adjusted according to the stage of training and the improvement of prediction performance to obtain the cargo image retrieval model.

[0055] According to an embodiment of the present invention, a cargo image to be retrieved is obtained; the cargo image to be retrieved is input into a cargo image retrieval model to obtain cargo text data corresponding to the cargo image to be retrieved; the cargo image retrieval model is obtained by constructing a multimodal regularization module, and multimodal feature fusion training is performed on text features generated by a text encoder for cargo text sample data, and on image features generated by an image encoder for cargo image sample data, wherein the loss function in the training includes a triplet loss function, and the marginal value of the triplet loss function is dynamically adjusted according to the stage of training and the improvement of prediction performance; the method provides a more comprehensive and rich feature expression by multimodal feature fusion of text features and image features through a multimodal regularization module, and at the same time, the marginal value of the triplet loss function is dynamically adjusted according to the stage of training and the improvement of prediction performance in combination with the training, thereby improving the matching accuracy between the cargo image and the corresponding cargo text, and improving the accuracy of cargo image retrieval.

[0056] As an optional implementation, the loss function in the training includes a triplet loss function, and the marginal value of the triplet loss function is dynamically adjusted according to the stage of training and the improvement of prediction performance, including:

[0057] The formula of the loss function is:

[0058] Loss = l(x a ,x p ,x n ,α)+l BCE

[0059] Among them, l(x a ,x p ,x n ,α) is the triple loss function, x a is the anchor point data, x p is the positive sample data, x n is the negative sample data, α is the marginal value; l BCEis a binary cross entropy loss function, which calculates whether the text feature and the image feature match, and is used for training the multimodal regularization module;

[0060] The marginal value of the triplet loss function is adjusted according to the stage of training, the verification accuracy of the current time step, and the boost coefficient of different training stages and the marginal value dynamic adjustment formula determined by the preset boost threshold.

[0061] The loss function consists of two parts: the triplet loss function and the binary cross entropy loss function. The triplet loss function can make similar sample data closer in the feature space and dissimilar sample data more dispersed. The binary cross entropy loss function is used for training the multimodal regularization module. That is, through the binary cross entropy loss function, the model can integrate the cargo image sample data and cargo text sample data from different modalities, calculate whether the image features corresponding to the cargo image sample data and the text features corresponding to the cargo text sample data match, and make effective classification predictions.

[0062] The formula of the loss function is:

[0063] Loss = l(x a ,x p ,x n ,α)+l BCE

[0064] Among them, l(x a ,x p ,x n ,α) is the triple loss function; l BCE is the binary cross entropy loss function.

[0065] The formula for the triplet loss function is:

[0066] l(x a ,x p ,x n ,α)=[d(x a ,x p )+α-d(x a ,x n )]+

[0067] Among them, x a 、x p 、x n are anchor points, positive samples and negative samples respectively. The anchor points are samples to be retrieved, positive samples are similar samples, and negative samples are dissimilar samples. α is the preset boundary value. d(x a ,x p ) is the distance function between the anchor point and the positive sample, d(x a ,xn ) is the distance function between the anchor point and the negative sample, [·]+ represents a non-negative boundary, ensuring that the distance between the positive sample and the negative sample differs by at least α, and the goal is to make the distance between the anchor point and the positive sample smaller than the distance between the anchor point and the negative sample by at least one marginal value, which helps the cargo image retrieval model learn to distinguish between positive and negative samples in different modalities. The marginal value α of the triplet loss function is adjusted according to the stage of training, the verification accuracy of the current time step, and the boost coefficient of different training stages and the marginal value dynamic adjustment formula determined by the preset boost threshold, which can improve the matching accuracy of cargo images and cargo text data, thereby improving the accuracy of cargo image retrieval in combination with the multimodal regularization module constructed in the model.

[0068] The binary cross entropy loss function in the loss function calculates whether the sample features of the cargo text and the sample features of the cargo image match, so as to optimize the multimodal feature fusion training of the multimodal regularization module in the image retrieval model on the multimodal feature fusion of the cargo text sampling features and the cargo image sampling features, so as to improve the accuracy of cargo image retrieval.

[0069] In the embodiment of the present invention, the marginal value of the triplet loss function in the loss function is adjusted according to the stage of training, the verification accuracy of the current time step, and the boost coefficient of different training stages and the marginal value dynamic adjustment formula determined by the preset boost threshold. By optimizing the training of the cargo image retrieval model, the accuracy of the cargo image retrieval model for cargo image retrieval is improved.

[0070] As an optional implementation manner, the marginal value dynamic adjustment formula includes:

[0071]

[0072] Among them, α t+1 is the marginal value at time step t+1, α t is the marginal value at time step t, Δp t is the verification accuracy p of the current time step t+1 Compared with the verification accuracy p of the previous time step t , Δp1 is the preset performance improvement threshold in the early stage of training, Δp2 is the preset performance improvement threshold in the middle stage of training, Δp3 is the preset performance improvement threshold in the late stage of training, c1 is the improvement coefficient of the marginal value increase in the early stage of training, c2 is the improvement coefficient of the marginal value increase in the middle stage of training, c3 is the improvement coefficient of the marginal value increase in the late stage of training, and Δp1<Δp2<Δp3, c1>c2>c3.

[0073] The early, middle and late stages of training can be defined by performance improvement, measured by the improvement in verification accuracy. In an optional embodiment, the early, middle and late stages of training are defined by training time, such as the number of training rounds. The specific values ​​of c1, c2 and c3 can be adjusted according to the actual model training situation to ensure that the dynamic adjustment of the marginal value α can not only promote the early learning of the model, but also ensure that the training difficulty is not excessively increased in the middle and late stages of training, which not only avoids falling into overfitting, improves the accuracy of model prediction, but also reduces unnecessary waste of computing power.

[0074] The marginal value is dynamically adjusted through the marginal value dynamic adjustment formula. In the early stage of cargo image retrieval model training, the task is relatively difficult and the model may not have learned enough features. Therefore, the goal of this stage is to let the model learn quickly and capture the basic distribution of the data. The preset performance improvement threshold Δp1 in the early stage of training can be set low so that the model can adjust the marginal value when the performance is slightly improved, encouraging the model to learn quickly. At the same time, the improvement coefficient c1 of the marginal value increase in the early stage of training can be large to quickly increase the training difficulty and prompt the model to quickly adapt to the data characteristics; when the cargo image retrieval model has learned certain features and the performance improvement begins to slow down, the model training enters the mid-term training stage. The preset performance improvement threshold Δp2 in the mid-term training stage can be moderately increased, because the model needs a greater performance improvement to trigger the adjustment of the marginal value. At the same time, the improvement coefficient c2 of the marginal value increase in the mid-term training stage can be moderate to maintain the stability of the model training while continuing to improve the model's discrimination ability; in the later stage of training, the cargo image retrieval model is relatively mature. At this time, the goal is to fine-tune the model to improve its generalization ability and avoid overfitting. The preset performance improvement threshold Δp3 in the later stages of training can be set higher. The margin value is adjusted only when the model has a significant performance improvement to maintain the stability of the model. At the same time, the improvement coefficient c3 of the marginal value increase in the later stages of training may be small, or even gradually decrease. The marginal value is fine-tuned to ensure the generalization ability of the model on new data. As training progresses, the marginal value gradually increases and eventually reaches a maximum value.

[0075] The embodiments of the present invention dynamically adjust the marginal value so that the training difficulty can be adaptively adjusted during the cargo image retrieval model training, thereby avoiding the optimization difficulties that may be caused by fixed boundary values, improving the accuracy of model prediction, and also improving the efficiency of training.

[0076] As an optional implementation, the cargo image retrieval method further includes:

[0077] The text encoder includes a hierarchical Transformer encoder and a first Transformer decoder constructed by combining a self-attention mechanism and a mutual attention mechanism;

[0078] The hierarchical Transformer encoder encodes the sender information, address information and cargo information in the cargo text sample data, and the encoded information is processed by the first Transformer decoder to generate the text features.

[0079] The text encoder includes a hierarchical Transformer encoder and a first Transformer decoder constructed by combining the self-attention mechanism and the mutual attention mechanism. During the training of the cargo image retrieval model, after the cargo text sample data including the sender information, the shipping address information, the receiving address information, the cargo type information and the cargo weight information are input into the text encoder, the hierarchical Transformer encoder first encodes the cargo text data, and the encoded information is captured by the first Transformer decoder to generate the text features after capturing the interactive relationship between different descriptive entities in the cargo text data.

[0080] In an embodiment of the present invention, a hierarchical Transformer encoder is used to encode the sender information, address information and cargo information in the cargo text sample data. The encoded information is processed by a first Transformer decoder to generate text features, which can capture the interactive relationship between different descriptive entities in the cargo text data, so that the text features have deeper semantic information, enhance the semantic understanding ability of the model, and thus improve the accuracy of the model prediction.

[0081] As an optional implementation, the hierarchical Transformer encoder encodes the sender information, address information and cargo information in the cargo text sample data, and the encoded information is processed by the first Transformer decoder to generate the text features, including:

[0082] The first Transformer encoder in the hierarchical Transformer encoder respectively performs the first encoding on the sender information, the address information and the cargo information, and correspondingly outputs the first encoded sender information, the first encoded address information and the first encoded cargo information;

[0083] The second Transformer encoder in the hierarchical Transformer encoder respectively performs a second encoding on the address information after the first encoding and the cargo information after the first encoding, and correspondingly outputs the address information after the second encoding and the cargo information after the second encoding;

[0084] One of the sender information after the first encoding, the address information after the second encoding, and the cargo information after the second encoding is used as a query, and the remaining two are input into the first Transformer decoder for processing as a combination of Key and Value, and the processed outputs are spliced ​​to obtain the text feature.

[0085] like Figure 3 As shown, it is a processing diagram of a text encoder, and the text editor includes a first Transformer encoder, a second Transformer encoder, and a first Transformer decoder D including three decoding units D1, D2, and D3. The sender information in the cargo text sample data is the shipping company, the address information includes the shipping address and the receiving address, and the cargo information includes the cargo type and cargo weight.

[0086] The sender company information is encoded by the first Transformer encoder, and the encoded sender information is recorded as T1; the sender address information and the recipient address information are encoded for the first time by the first Transformer encoder, and the encoded sender address information and the recipient address information are recorded as T21 and T22 respectively. T21 and T22 are further processed by the second Transformer encoder, and the second encoded address information is recorded as HT1; the cargo type information and cargo weight information are encoded for the first time by the first Transformer encoder, and the encoded cargo type information and cargo weight information are recorded as T31 and T32 respectively. T31 and T32 are further processed by the second Transformer encoder, and the second encoded cargo information is recorded as HT2.

[0087] Take T1 as Query, HT1 and HT2 as Key and Value and input them into D1 in the first Transformer decoder D for processing; take HT1 as Query, T1 and HT2 as Key and Value and input them into D2 in the first Transformer decoder D for processing; take HT2 as Query, T1 and HT1 as Key and Value and input them into D3 in the first Transformer decoder D for processing. The processed outputs of D1, D2 and D3 are concatenated to obtain the text features.

[0088] The embodiment of the present invention can capture the interactive relationship between different descriptive entities in the cargo text sample data by performing two encoding processes and one decoding process on the sender information, address information and cargo information in the cargo text sample data, and splicing the outputs obtained from the decoding process to obtain text features, so that the text features have deeper semantic information, enhance the semantic understanding ability of the model, and thus improve the accuracy of the model prediction.

[0089] As an optional implementation, the cargo image retrieval method further includes:

[0090] The multimodal regularization module includes a second Transformer decoder, which combines the cross-attention mechanism to perform inter-modal information fusion and regularization processing on the text features and image features.

[0091] The multimodal regularization module includes a second Transformer decoder, and combines the cross-attention mechanism to perform information fusion and regularization processing on text features and image features between modalities. It is only used in the model training stage. The multimodal regularization module obtains the Query (Q vector) from one of the modalities of image features and text features, and obtains the Key (K vector) and Value (V vector) from the remaining other modality, thereby realizing information fusion between modalities. Specifically, the Q vector obtained from one modality is obtained by calculating the similarity with the KV vector from the other modality to obtain the corresponding weight, and then the KV vector is weighted and summed by the weight to update the original Q vector, thereby realizing information fusion and regularization between modalities.

[0092] The embodiment of the present invention introduces a multimodal regularization module to enhance the learning ability of the cargo image retrieval model for the interaction between the two modalities corresponding to the image features and the text features during training, thereby providing a more comprehensive and rich feature expression, which not only improves the prediction accuracy of the cargo image retrieval model, but also improves the efficiency.

[0093] As an optional implementation, the cargo image retrieval method further includes:

[0094] The visual Transformer is used as the image encoder to convert the cargo image sample data into a series of image Tokens to generate the image features.

[0095] In some embodiments, the visual transformer may be ViT (Vision Transformer) or CLIP-ViT. By using ViT or CLIP-ViT as the image encoder, the cargo image sample data may be converted into a series of image Tokens to generate the image features.

[0096] In the embodiment of the present invention, by using a visual Transformer as the image encoder, the sample data of the cargo image is converted into a series of image Tokens to generate image features, which can improve the accuracy of image feature extraction and further improve the accuracy of object image retrieval.

[0097] Example 2

[0098] Based on the above-mentioned cargo image retrieval method, the embodiment of the present invention further provides a cargo image retrieval device, the structural diagram of which is as follows: Figure 4 As shown, the cargo image retrieval device 40 includes a data acquisition module 41 and a retrieval module 42;

[0099] The data acquisition module 41 acquires the image of the goods to be retrieved;

[0100] The retrieval module 42 is used to input the cargo image to be retrieved into the cargo image retrieval model to obtain cargo text data corresponding to the cargo image to be retrieved; the cargo image retrieval model is obtained by constructing a multimodal regularization module, and performing multimodal feature fusion training on the text features generated by the text encoder for the cargo text sample data, and the image features generated by the image encoder for the cargo image sample data. The loss function in the training includes a triple loss function, and the marginal value of the triple loss function is dynamically adjusted according to the stage of training and the improvement of prediction performance.

[0101] For other details about how the modules in the cargo image retrieval device implement the above technical solution, please refer to the description of the cargo image retrieval method provided in the above invention embodiment, which will not be repeated here.

[0102] Example 3

[0103] Based on the above-mentioned cargo image retrieval method, the embodiment of the present invention further provides a cargo image retrieval device, the structural diagram of which is as follows: Figure 5 As shown, the cargo image retrieval device 50 includes a processor 51 and a memory 52 coupled to the processor 51. The memory 52 stores a computer program, and when the computer program is executed by the processor 51, the processor 51 executes the steps of the cargo image retrieval method in the above embodiment.

[0104] For other details about the processor 51 in the above-mentioned cargo image retrieval device 50 to implement the above-mentioned technical solution, please refer to the description of the cargo image retrieval device method provided in the above-mentioned invention embodiment, which will not be repeated here.

[0105] Among them, the processor 51 can also be called a CPU (Central Processing Unit), and the processor 51 may be an integrated circuit chip with signal processing capabilities; the processor 51 can also be a general-purpose processor, DSP (Digital Signal Process), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, among which the general-purpose processor can be a microprocessor or the processor 51 can also be any conventional processor, etc.

[0106] Example 4

[0107] The embodiment of the present invention further provides a computer-readable storage medium, a schematic diagram of which is shown in FIG. Figure 6 As shown, a readable computer program 61 is stored on the storage medium; wherein the computer program 61 can be stored in the above storage medium in the form of a software product, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, magnetic disk or optical disk, ROM (Read-Only Memory), RAM (Random Access Memory) and other media that can store program codes, or terminal devices such as computers, servers, mobile phones, and tablets.

[0108] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0109] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0110] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0111] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0112] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0113] The technical solution provided by the present application is introduced in detail above. The principles and implementation methods of the present application are explained by using specific examples in the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0114] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0115] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0116] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0118] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A cargo image retrieval method, characterized in that: include: Get the pictures of the goods to be retrieved; The cargo picture to be retrieved is input into a cargo picture retrieval model to obtain cargo text data corresponding to the cargo picture to be retrieved; the cargo picture retrieval model is obtained by constructing a multimodal regularization module, and performing multimodal feature fusion training on text features generated by a text encoder for cargo text sample data, and on image features generated by an image encoder for cargo picture sample data, wherein the loss function in the training includes a triplet loss function, and the marginal value of the triplet loss function is dynamically adjusted according to the stage of training and the improvement of prediction performance.

2. The cargo image retrieval method according to claim 1, characterized in that: The loss function in the training includes a triplet loss function, and the marginal value of the triplet loss function is dynamically adjusted according to the stage of training and the improvement of prediction performance, including: The formula of the loss function is Loss=l(x a ,x p ,x n ,α)+l BCE Among them, l(x a ,x p ,x n ,α) is the triple loss function, x a is the anchor point data, x p is the positive sample data, x n is the negative sample data, α is the marginal value; l BCE is a binary cross entropy loss function, which calculates whether the text feature and the image feature match, and is used for training the multimodal regularization module; The marginal value of the triplet loss function is adjusted according to the stage of training, the verification accuracy of the current time step, and the boost coefficient of different training stages and the marginal value dynamic adjustment formula determined by the preset boost threshold.

3. The cargo image retrieval method according to claim 2, characterized in that: The marginal value dynamic adjustment formula includes: Among them, α t+1 is the marginal value at time step t+1, α t is the marginal value at time step t, Δp t is the verification accuracy p of the current time step t+1 Compared with the verification accuracy p of the previous time step t , Δp1 is the preset performance improvement threshold in the early stage of training, Δp2 is the preset performance improvement threshold in the middle stage of training, Δp3 is the preset performance improvement threshold in the late stage of training, c1 is the improvement coefficient of the marginal value increase in the early stage of training, c2 is the improvement coefficient of the marginal value increase in the middle stage of training, c3 is the improvement coefficient of the marginal value increase in the late stage of training, and Δp1<Δp2<Δp3, c1>c2>c3.

4. The cargo image retrieval method according to claim 1, characterized in that: The cargo image retrieval method further includes: The text encoder includes a hierarchical Transformer encoder and a first Transformer decoder constructed by combining a self-attention mechanism and a mutual attention mechanism; The hierarchical Transformer encoder encodes the sender information, address information and cargo information in the cargo text sample data, and the encoded information is processed by the first Transformer decoder to generate the text features.

5. The cargo image retrieval method according to claim 4, characterized in that: The hierarchical Transformer encoder encodes the sender information, address information and cargo information in the cargo text sample data, and the encoded information is processed by the first Transformer decoder to generate the text features, including: The first Transformer encoder in the hierarchical Transformer encoder respectively performs the first encoding on the sender information, the address information and the cargo information, and correspondingly outputs the first encoded sender information, the first encoded address information and the first encoded cargo information; The second Transformer encoder in the hierarchical Transformer encoder respectively performs a second encoding on the address information encoded for the first time and the cargo information encoded for the first time, and correspondingly outputs the address information encoded for the second time and the cargo information encoded for the second time; One of the sender information after the first encoding, the address information after the second encoding, and the cargo information after the second encoding is used as a query, and the remaining two are input into the first Transformer decoder for processing as a combination of Key and Value, and the processed outputs are spliced ​​to obtain the text feature.

6. The cargo image retrieval method according to claim 1, characterized in that: The cargo image retrieval method further includes: The multimodal regularization module includes a second Transformer decoder, which combines the cross-attention mechanism to perform inter-modal information fusion and regularization processing on the text features and image features.

7. The cargo image retrieval method according to claim 1, characterized in that: The cargo image retrieval method further includes: The visual Transformer is used as the image encoder to convert the cargo image sample data into a series of image Tokens to generate the image features.

8. A cargo image retrieval device, characterized in that: It includes a data acquisition module and a retrieval module; The data acquisition module acquires the image of the goods to be retrieved; The retrieval module is used to input the cargo image to be retrieved into the cargo image retrieval model to obtain cargo text data corresponding to the cargo image to be retrieved; the cargo image retrieval model is obtained by constructing a multimodal regularization module, performing multimodal feature fusion training on the text features generated by the cargo text sample data via a text encoder, and the image features generated by the cargo image sample data via an image encoder, wherein the loss function in the training includes a triplet loss function, and the marginal value of the triplet loss function is dynamically adjusted according to the stage of training and the improvement of prediction performance.

9. A cargo image retrieval device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store computer programs; The processor is used to read the computer program in the memory and execute the steps of the cargo image retrieval method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A readable computer program is stored thereon, and when the program is executed by a processor, the steps of the cargo image retrieval method as described in any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Cross-modal retrieval method and device, computer equipment and storage medium

    CN109783655A

  • Image-text cross-modal retrieval method, system and device and storage medium

    CN110059157A

  • Cross-modal retrieval method, device and equipment and computer readable storage medium

    CN112487217A

  • Cross-modal retrieval model training method, cross-modal retrieval method, equipment and medium

    CN114841243A