Commodity matching method and device

Abstract

The invention discloses a commodity matching method and device, and relates to the technical field of computers. A specific implementation mode of the method comprises the steps of obtaining picture data and initial waybill data of a to-be-matched commodity, inputting the picture data into a model to obtain description information of the corresponding to-be-matched commodity, performing matching based on the description information to determine a suspected waybill, and further, determining the to-be-matched commodity according to the suspected waybill. And determining a waybill matching result with the corresponding to-be-matched commodity in the suspected waybills by calculating the similarity. According to the embodiment, accurate matching of the parcels losing the waybill information can be realized, and the matching efficiency of the parcels and the waybills is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a commodity matching method and device. Background Art

[0002] At present, in the field of cargo transportation, due to the different origins and destinations, it is often necessary to regroup the goods for subsequent centralized transportation. In the process of regrouping, the goods need to be sorted at the sorting site. During the sorting process, due to the outer packaging of the goods, the mode of transportation, etc., the waybill information attached to the surface of the goods may fall off during the sorting operation. The waybill records the waybill number, destination and other important information of the goods. The "three-no naked goods" without the waybill cannot accurately locate their waybill information, and thus the goods cannot be transported.

[0003] At present, the determination of "three-no naked goods" information mainly relies on the visual judgment of the cargo data by site staff. The identified information is manually entered into the system and manually checked against the suspected waybills to finally determine the possible waybills. This is inefficient and has low accuracy. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method for matching goods, which can obtain description information based on the image data of the goods, further calculate the similarity with the waybill, determine the waybill matching result, and realize accurate identification of the goods waybill.

[0005] To achieve the above object, according to a first aspect of an embodiment of the present invention, a method for matching products is provided, comprising:

[0006] Get the image data and initial waybill data of the product to be matched;

[0007] Input the image data of the to-be-matched product into the model to obtain description information of the to-be-matched product;

[0008] Based on the description information, matching is performed with the initial waybill data to obtain a suspected waybill;

[0009] Calculating similarity based on the description information and the description information of the commodity corresponding to the suspected waybill;

[0010] The matching result of the to-be-matched product is determined based on the result of the similarity calculation.

[0011] Optionally, inputting the image data of the to-be-matched product into the model to obtain the description information of the to-be-matched product includes:

[0012] Inputting the image data of the to-be-matched product into the recognition model to obtain the text information of the to-be-matched product;

[0013] Inputting the image data of the to-be-matched product into the multimodal macro model to obtain appearance description information of the to-be-matched product;

[0014] The text information of the to-be-matched product and the appearance description information of the to-be-matched product are integrated to obtain the description information of the to-be-matched product.

[0015] Optionally, inputting the image data of the to-be-matched product into a recognition model to obtain text information of the to-be-matched product includes:

[0016] Input the image data of the product to be matched into the recognition model to obtain output image feature information;

[0017] The output image feature information is cleaned to obtain text information of the product to be matched.

[0018] Optionally, before inputting the image data of the to-be-matched product into the multimodal macro model to obtain the appearance description information of the to-be-matched product, the method further includes:

[0019] Acquire data of historically successfully matched commodities, wherein the data of historically successfully matched commodities include text information, image data of historically to-be-matched commodities, and description information of commodities corresponding to the historically successfully matched waybills;

[0020] Pre-training the multimodal large model based on the text information and image data of the historical products to be matched;

[0021] The parameters of the multimodal large model are corrected based on the description information of the goods corresponding to the historical waybills that have been successfully matched.

[0022] Optionally, the fusing the text information of the to-be-matched product and the appearance description information of the to-be-matched product to obtain the description information of the to-be-matched product includes:

[0023] The text information of the to-be-matched product and the appearance description information of the to-be-matched product are input into a natural language processing model to obtain the description information of the to-be-matched product.

[0024] Optionally, the calculating a similarity based on the description information and the description information of the commodity corresponding to the suspected waybill includes:

[0025] Performing vectorization processing on the description information and the description information of the commodity corresponding to the suspected waybill respectively;

[0026] Similarity calculation is performed based on the vectorized processing results.

[0027] Optionally, determining the matching result of the to-be-matched product based on the result of the similarity calculation includes:

[0028] Based on the result of the similarity calculation, the commodities corresponding to the suspected waybills are sorted in descending order of similarity to obtain an ordered sequence;

[0029] A matching result of the to-be-matched commodities is determined based on the ordered sequence.

[0030] According to a second aspect of an embodiment of the present invention, there is provided a commodity matching device, comprising:

[0031] The acquisition module is used to obtain the image data and initial waybill data of the product to be matched;

[0032] An input module, used to input the image data of the to-be-matched product into the model to obtain the description information of the to-be-matched product;

[0033] A matching module, configured to match the description information with the initial waybill data to obtain a suspected waybill;

[0034] A calculation module, configured to calculate a similarity between the description information and the description information of the commodity corresponding to the suspected waybill;

[0035] A determination module is used to determine the matching result of the to-be-matched product based on the result of the similarity calculation.

[0036] Optionally, the input module is used to input the image data of the to-be-matched product into the model to obtain the description information of the to-be-matched product, including:

[0037] The input module is used to input the image data of the to-be-matched product into the recognition model to obtain the text information of the to-be-matched product; input the image data of the to-be-matched product into the multimodal large model to obtain the appearance description information of the to-be-matched product;

[0038] The fusion module is used to fuse the text information of the to-be-matched product and the appearance description information of the to-be-matched product to obtain the description information of the to-be-matched product.

[0039] Optionally, the input module is used to input the image data of the to-be-matched product into the recognition model to obtain the text information of the to-be-matched product, including:

[0040] The input module is used to input the image data of the to-be-matched product into the recognition model to obtain output image feature information;

[0041] The cleaning module is used to clean the output image feature information to obtain the text information of the product to be matched.

[0042] Optionally, the device further comprises:

[0043] The acquisition module is further used to acquire data of commodities that have been matched successfully in history, wherein the data of commodities that have been matched successfully in history include text information and image data of commodities to be matched in history and description information of commodities corresponding to the waybills that have been matched successfully in history;

[0044] A training module, used for pre-training the multimodal large model based on the text information and image data of the historical products to be matched;

[0045] A correction module is used to correct the parameters of the multimodal large model based on the description information of the goods corresponding to the waybills that have been successfully matched in the history.

[0046] Optionally, the fusion module is used to fuse the text information of the to-be-matched product and the appearance description information of the to-be-matched product to obtain the description information of the to-be-matched product, including:

[0047] The fusion module is used to input the text information of the to-be-matched product and the appearance description information of the to-be-matched product into a natural language processing model to obtain the description information of the to-be-matched product.

[0048] Optionally, the calculation module is used to calculate the similarity based on the description information and the description information of the commodity corresponding to the suspected waybill, including:

[0049] A processing module, used to perform vectorization processing on the description information and the description information of the commodity corresponding to the suspected waybill respectively;

[0050] The calculation module is used to perform similarity calculation based on the vectorization processing result.

[0051] Optionally, the determining module is used to determine the matching result of the to-be-matched product based on the result of the similarity calculation, including:

[0052] A sorting module, configured to sort the commodities corresponding to the suspected waybills in descending order of similarity based on the result of the similarity calculation to obtain an ordered sequence;

[0053] The determination module is used to determine the matching result of the to-be-matched commodities based on the ordered sequence.

[0054] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0055] One or more processors; a storage device for storing one or more programs,

[0056] When the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the first aspect of the embodiment of the present invention.

[0057] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method provided by the first aspect of the embodiment of the present invention is implemented.

[0058] An embodiment of the above invention has the following advantages or beneficial effects: the description information of the goods to be matched is identified through a model, and matched with the waybill data, and the similarity between the description information and the waybill is further calculated, and the waybill matching result data of the goods to be matched is determined based on the similarity, thereby improving the accuracy of the waybill matching of the goods to be matched, and at the same time improving the matching efficiency, which helps to achieve the rapid positioning of "three-no naked goods".

[0059] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a schematic diagram of the main process of the commodity matching method according to an embodiment of the present invention;

[0061] Figure 2 It is a schematic diagram of a "three-no naked goods" provided according to an embodiment of the present invention;

[0062] Figure 3 is a schematic diagram of another “three-no naked goods” provided according to an embodiment of the present invention;

[0063] Figure 4 is a schematic diagram of a commodity matching process provided according to an embodiment of the present invention;

[0064] Figure 5 It is a schematic diagram of main modules of a commodity matching device provided by an embodiment of the present invention;

[0065] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0066] Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0068] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0069] Figure 1 A commodity matching method according to an embodiment of the present invention is as follows. Figure 1 As shown, including:

[0070] Step S101, obtaining image data and initial waybill data of the product to be matched.

[0071] Obtain all the image data of "three-no naked goods" to be matched reported by the user to the system, specifically, the images of "three-no naked goods" taken by the user through a device with a camera. Obtain the initial waybill data of the goods received by the current site. Specifically, the initial waybill obtained can be the waybill in the target state among the waybills obtained by the current site, and the target state can be the unsent state.

[0072] Optionally, the initial waybill data obtained may be waybill data for a target time period. For example, if the image of the commodity to be matched is identified at time a, the initial waybill obtained should be waybill b for the batch of goods being processed at time a.

[0073] Optionally, the acquired image data of “three-no naked goods” is image data identified from naked goods information reported by the user after the user performs a naked goods reporting operation.

[0074] Optionally, "three-no bare goods" refers to goods with missing delivery bills or damaged delivery bill information, such as Figure 2 and Figure 3 The goods shown. Figure 2 The goods shown do not have text information but have clear shapes and obvious materials, such as clothes, shoes, etc. Figure 3 As shown, the actual shape of the goods wrapped in packaging (cardboard boxes, plastic bags, etc.) cannot be seen, but the name of the goods, brand name and other information (xx brand egg yolk crisps, etc.) are attached on the outer packaging of the goods.

[0075] Step S102: input the image data of the to-be-matched product into the model to obtain the description information of the to-be-matched product.

[0076] The obtained image data of the product to be matched is input into the trained recognition model to obtain the output description information based on the product image. The description information may include information about the appearance style of the identified product and text information that can be recognized in the product image, such as the product's color, brand, name and other data.

[0077] Step S103, based on the description information, matching is performed with the initial waybill data to obtain a suspected waybill.

[0078] According to the description information of the to-be-matched goods output by the model, fuzzy matching is performed with the initial waybill to obtain a certain number of waybills as suspected waybills. That is, a certain number of suspected waybills are determined based on the degree of matching between the description information and the waybill product information, which are used for the subsequent determination of matching waybill data.

[0079] Optionally, when performing fuzzy matching of the initial waybill based on the description information, fuzzy matching can be performed according to the information corresponding to the target attribute in the description information. For example, matching can be performed using the product name in the description information, or matching can be performed based on the product brand name, etc. The selection of specific target attributes can be set according to actual conditions and is not limited to this.

[0080] Step S104, performing similarity calculation based on the description information and the description information of the commodity corresponding to the suspected waybill.

[0081] The description information of the commodity corresponding to the suspected commodity is determined based on the acquired data of the suspected waybill, and the similarity is calculated based on the description information of the corresponding commodity and the description information of the commodity to be matched.

[0082] Optionally, before calculating the similarity, two pieces of description information are first vectorized, and the similarity is calculated based on the processing result.

[0083] Step S105: determining the matching result of the to-be-matched product based on the result of the similarity calculation.

[0084] The target matching waybill in the suspected waybill is determined according to the calculated similarity between the to-be-matched product and the product corresponding to each suspected waybill, and the product corresponding to the target matching waybill is determined as the target matching product. The target matching waybill may be at least one waybill, and the corresponding target matching product may also be at least one product.

[0085] Optionally, determining the matching result of the commodity to be matched based on the result of similarity calculation includes: based on the result of similarity calculation, sorting the commodities corresponding to the suspected waybills in order of similarity from high to low to obtain an ordered sequence; and determining the matching result of the commodity to be matched based on the ordered sequence. The determined target matching waybills may be an ordered sequence of suspected waybills arranged from large to small similarity, corresponding to an ordered sequence of commodities from large to small similarity. According to the ordered sequence, it is determined in sequence whether it is the target commodity corresponding to the commodity to be matched.

[0086] Optionally, the determined target matching waybill may also be a list of suspected waybills whose similarity values ​​are greater than a preset threshold, and the suspected waybill data in the list is randomly obtained and matched with the product to be matched to determine whether it is the target product corresponding to the product to be matched.

[0087] According to the commodity matching method provided by the embodiment of the present invention, the text information and appearance description information of the image data of the commodity to be matched can be obtained through the model, and the description information of the commodity to be matched can be obtained by fusion, and the suspected waybill can be matched based on the description information. On this basis, the matching result data of the commodity to be matched can be finally determined by calculating the similarity between the description information of the commodity to be matched and the description information of the suspected waybill, and then the waybill corresponding to the commodity to be matched can be determined, so as to realize the automatic matching of the commodity to be matched and the waybill, and improve the matching efficiency of the commodity and the waybill. In addition, the similarity calculation based on the commodity image data can improve the accuracy of matching.

[0088] Optionally, the image data of the product to be matched is input into the model to obtain description information of the product to be matched, including: inputting the image data of the product to be matched into the recognition model to obtain text information of the product to be matched; inputting the image data of the product to be matched into the multimodal large model to obtain appearance description information of the product to be matched; fusing the text information of the product to be matched and the appearance description information of the product to be matched to obtain description information of the product to be matched.

[0089] When generating the description information of the product based on the image data of the product to be matched, the image data is first input into the trained optical character recognition (OCR) model to obtain the text information of the product to be matched in the image output by the model. If the image of the product to be matched does not contain text information, the output text information is empty. Then the image data of the product to be matched is input into the trained multimodal large model, and the model outputs the appearance description information based on the understanding of the image according to the pre-set question sentence. Here, the output data of the model is illustrated by an example. The pre-set question sentence can be "What is this? Describe the picture", and the output of the model based on the question sentence can be "This is a rectangular block with a yellow paper box outer packaging, with a cookie pattern on it, and inside is a rectangular block similar to stainless steel material, which should be an iron box.", "This is a black winter coat, it looks very thick, and may be made of wool." and other similar product appearance description information.

[0090] The text information output by the acquired OCR model and the appearance description information output by the multimodal model are fused to obtain the key information of the product to be matched, i.e., the description information. Specifically, through the fusion process, important keyword data in the two data are retained, such as brand, product name, weight, etc.

[0091] Optionally, inputting the image data of the to-be-matched product into the recognition model to obtain the text information of the to-be-matched product includes: inputting the image data of the to-be-matched product into the recognition model to obtain output image feature information; cleaning the output image feature information to obtain the text information of the to-be-matched product. Inputting the image of the to-be-matched product into the OCR model to obtain feature information in the recognized image, that is, recognition information of all texts and symbols. Cleaning the feature information to remove useless information, such as punctuation marks, telephone numbers, etc., to obtain the final text information of the to-be-recognized product.

[0092] Optionally, when performing text cleaning, the more generalized text or the text with a high degree of overlap among similar products will also be cleaned and removed, such as data such as food ingredient information.

[0093] Optionally, before inputting the image data of the goods to be matched into the multimodal large model and obtaining the appearance description information of the goods to be matched, the method also includes: obtaining the data of the goods that have been matched successfully in the past, the data of the goods that have been matched successfully in the past including the text information, image data and description information of the goods corresponding to the waybills that have been matched successfully in the past; pre-training the multimodal large model based on the text information and image data of the goods to be matched in the past; and correcting the parameters of the multimodal large model based on the description information of the goods corresponding to the waybills that have been matched successfully in the past. Before outputting the appearance description information of the goods through the model, it is necessary to train the model according to the historical data to obtain a more accurate output result. Specifically, the data of the goods that have been matched successfully in the past are obtained, including the image data, text information and description information of the waybills, and the text information therein is segmented and encoded, and the image data is feature extracted. Specifically, the text information is encoded using the input format of BERT; for the image data, feature extraction is performed and converted into data in vector form. The BERT model is integrated into a unified multimodal large model, and a multimodal pre-training task is designed based on the text information, image data and description information of the waybills. The multimodal large model is trained using pre-training tasks and fine-tuned according to the task objectives. The task objectives are the description information of the waybill; the training stop condition of the model can be set to the similarity between the appearance description information of the product output by the model and the description information based on the waybill is greater than the preset similarity threshold.

[0094] Optionally, the text information of the to-be-matched product and the appearance description information of the to-be-matched product are fused to obtain the description information of the to-be-matched product, including: inputting the text information of the to-be-matched product and the appearance description information of the to-be-matched product into a natural language processing model to obtain the description information of the to-be-matched product. The data fusion operation is performed through the natural language processing (Natural Language Processing, NLP) model to fuse the text information and the appearance description information to obtain the key description information of the to-be-matched product. For example, for a "three-no naked product", the text information output by the OCR model is "OCR: Member'sMark Selected Butter Cookies, Net Content 630g (315g*2)", and the appearance description information output by the multimodal model is "This is a rectangular block with a yellow paper box outer packaging, with a cookie pattern on it, and inside is a rectangular block similar to stainless steel material, which should be an iron box." The fused data output by the NLP model is "A yellow paper box outer packaging, it is Member's Mark Selected Butter Cookies, Net Content 630g (315g*2)"; for another "three-no naked product", the text information output by the OCR model is "empty", that is, the text information is not extracted from the image, and the appearance description information output by the multimodal model is "This is a black winter coat, it looks very thick, probably made of wool", and the output result obtained through the NLP model is "black wool coat".

[0095] Optionally, performing similarity calculation based on the description information and the description information of the commodity corresponding to the suspected waybill includes: respectively performing vectorization processing on the description information and the description information of the commodity corresponding to the suspected waybill; and performing similarity calculation based on the vectorization processing result.

[0096] Before similarity calculation, the acquired data is first vectorized to obtain an accurate similarity calculation result. Specifically, the description information of the to-be-matched product and the description information of the product corresponding to the suspected waybill are vectorized to obtain a first vector and a second vector, and similarity is calculated based on the first vector and the second vector to obtain a calculation result.

[0097] Optionally, during the vectorization process before similarity calculation, other feature data of the goods to be matched can also be obtained, such as the batch number of the goods, the license plate number of the goods transportation, etc. The priority of this data is lower than the priority of the image data of the above goods. The other feature data of the goods to be matched obtained can also be vectorized, and accordingly, the data corresponding to the goods suspected of being waybills also need to be vectorized. For example, if the batch number of the goods to be matched is vectorized, then the batch number of the suspected waybill also needs to be vectorized, and the similarity is calculated based on the vectorization result.

[0098] According to the commodity matching method provided by the embodiment of the present invention, the text information and appearance description information of the image data of the commodity to be matched can be obtained through the OCR model and the multimodal large model, and the description information of the commodity to be matched can be obtained by fusion, and the suspected waybill can be matched based on the description information. The matching result data of the commodity to be matched can be finally determined by similarity calculation, and then the waybill corresponding to the commodity to be matched can be determined, so that the automatic matching of the commodity to be matched and the waybill is realized, and the matching efficiency of the commodity to be matched is improved. In addition, the similarity calculation based on the commodity image data can improve the accuracy of matching, which is helpful to realize the fast and accurate waybill positioning of "three-no naked goods".

[0099] like Figure 4 The figure shows a schematic diagram of a product matching process. As shown in the figure, the data of the three-no naked goods reported by the staff is obtained, and the image data is input into the OCR model and the multimodal large model to obtain text information and appearance description information respectively, and then vectorized. At the same time, other features of the three-no naked goods can be obtained for vectorization, and the similarity between the vectorization results and the vectorization results of the corresponding data in the suspected waybill is calculated to determine the waybill sequence with the highest matching degree. Among them, the highest matching degree means the largest similarity value.

[0100] Figure 5 1 is a schematic diagram of main modules of a commodity matching device 500 provided according to an embodiment of the present invention, including:

[0101] The acquisition module 501 is used to acquire the image data and initial waybill data of the product to be matched;

[0102] An input module 502 is used to input the image data of the to-be-matched product into the model to obtain the description information of the to-be-matched product;

[0103] A matching module 503 is used to match the description information with the initial waybill data to obtain a suspected waybill;

[0104] A calculation module 504, configured to calculate similarity based on the description information and the description information of the commodity corresponding to the suspected waybill;

[0105] The determination module 505 is used to determine the matching result of the to-be-matched commodity based on the result of the similarity calculation.

[0106] According to the device for matching goods provided by the embodiment of the present invention, the text information and appearance description information of the image data of the goods to be matched can be obtained through the model, and the description information of the goods to be matched can be obtained by fusion, and the suspected waybill can be matched based on the description information. On this basis, the matching result data of the goods to be matched can be finally determined by calculating the similarity between the description information of the goods to be matched and the description information of the suspected waybill, and then the waybill corresponding to the goods to be matched can be determined, so that the automatic matching of the goods to be matched and the waybill is realized, and the matching efficiency of the goods and the waybill is improved. In addition, the similarity calculation based on the goods image data can improve the accuracy of matching.

[0107] Optionally, the input module 502 is used to input the image data of the to-be-matched product into the model to obtain the description information of the to-be-matched product, including:

[0108] Input module 502, used to input the image data of the to-be-matched product into the recognition model to obtain the text information of the to-be-matched product; input the image data of the to-be-matched product into the multimodal large model to obtain the appearance description information of the to-be-matched product;

[0109] The fusion module 506 is used to fuse the text information of the to-be-matched product and the appearance description information of the to-be-matched product to obtain the description information of the to-be-matched product.

[0110] Optionally, the input module 502 is used to input the image data of the to-be-matched product into the recognition model to obtain the text information of the to-be-matched product, including:

[0111] An input module 502 is used to input the image data of the product to be matched into the recognition model to obtain output image feature information;

[0112] The cleaning module 507 is used to clean the output image feature information to obtain text information of the product to be matched.

[0113] Optionally, the device further comprises:

[0114] The acquisition module 501 is also used to acquire data of historically successfully matched commodities, where the data of historically successfully matched commodities include text information, image data of historically to-be-matched commodities, and description information of commodities corresponding to historically successfully matched waybills;

[0115] A training module 508 is used to pre-train the multimodal large model based on the text information and image data of historical products to be matched;

[0116] The correction module 509 is used to correct the parameters of the multimodal large model based on the description information of the goods corresponding to the waybills that have been successfully matched in the past.

[0117] Optionally, the fusion module 506 is used to fuse the text information of the to-be-matched product and the appearance description information of the to-be-matched product to obtain the description information of the to-be-matched product, including:

[0118] The fusion module 506 is used to input the text information of the to-be-matched product and the appearance description information of the to-be-matched product into the natural language processing model to obtain the description information of the to-be-matched product.

[0119] Optionally, the calculation module 504 is used to perform similarity calculation based on the description information and the description information of the commodity corresponding to the suspected waybill, including:

[0120] A processing module 510 is used to perform vectorization processing on the description information and the description information of the commodity corresponding to the suspected waybill respectively;

[0121] The calculation module 504 is used to perform similarity calculation based on the vectorization processing result.

[0122] Optionally, the determination module 505 is used to determine the matching result of the to-be-matched product based on the result of the similarity calculation, including:

[0123] The sorting module 511 is used to sort the commodities corresponding to the suspected waybills in descending order of similarity based on the result of similarity calculation to obtain an ordered sequence;

[0124] The determination module 505 is used to determine the matching results of the commodities to be matched based on the ordered sequence.

[0125] According to the device for matching goods provided by the embodiment of the present invention, the text information and appearance description information of the image data of the goods to be matched can be obtained through the OCR model and the multimodal large model, and the description information of the goods to be matched can be obtained by fusion, and the suspected waybill can be matched based on the description information. The matching result data of the goods to be matched can be finally determined by similarity calculation, and then the waybill corresponding to the goods to be matched can be determined, so that the automatic matching of the goods to be matched and the waybill is realized, and the matching efficiency of the goods to be matched is improved. In addition, the similarity calculation based on the product image data can improve the accuracy of matching, which is helpful to realize the fast and accurate waybill positioning of "three-no naked goods".

[0126] Figure 6 An exemplary system architecture 600 is shown to which the collaborative modeling method or collaborative modeling apparatus according to an embodiment of the present invention may be applied.

[0127] like Figure 6As shown, system architecture 600 may include terminal devices 601, 602, 603, network 604 and server 605. Network 604 is used to provide a medium for communication links between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0128] The user can use the terminal devices 601, 602, 603 to interact with the server 605 through the network 604 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 601, 602, 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0129] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0130] The server 605 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 601, 602, and 603. The backend management server may analyze and process the received data such as the modeling request, and feed back the processing result (such as the target model - only an example) to the terminal device.

[0131] It should be noted that the collaborative modeling method provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the collaborative modeling device is generally set in the server 605 .

[0132] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0133] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0134] like Figure 7As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0135] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0136] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0137] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0138] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0139] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes an acquisition module, a fusion module, an input module, a determination module and an update module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the acquisition module may also be described as a "module for acquiring raw data and information about the population sent by the generator".

[0140] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0141] Get the image data and initial waybill data of the product to be matched;

[0142] Input the image data of the product to be matched into the model to obtain the description information of the product to be matched;

[0143] Based on the description information, match it with the initial waybill data to obtain the suspected waybill;

[0144] Calculate the similarity between the description information and the description information of the product corresponding to the suspected waybill;

[0145] The matching result of the to-be-matched product is determined based on the result of the similarity calculation.

[0146] According to the technical solution of the embodiment of the present invention, the text information and appearance description information of the image data of the to-be-matched goods can be obtained through the OCR model and the multimodal large model, and the description information of the to-be-matched goods can be obtained by fusion, and the suspected waybill can be matched based on the description information. And through similarity calculation, the matching result data of the to-be-matched goods can be finally determined, and then the waybill corresponding to the to-be-matched goods can be determined, so as to realize the automatic matching of the to-be-matched goods and the waybill, and improve the matching efficiency of the to-be-matched goods. In addition, the similarity calculation based on the product image data can improve the accuracy of matching, which is helpful to realize the fast and accurate waybill positioning of "three-no naked goods".

[0147] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for matching products. It is characterized in that include: Get the image data and initial waybill data of the product to be matched; Input the image data of the to-be-matched product into the model to obtain description information of the to-be-matched product; Based on the description information, matching is performed with the initial waybill data to obtain a suspected waybill; Calculating similarity based on the description information and the description information of the commodity corresponding to the suspected waybill; The matching result of the to-be-matched product is determined based on the result of the similarity calculation.

2. The method according to claim 1, It is characterized in that The step of inputting the image data of the to-be-matched product into the model to obtain the description information of the to-be-matched product includes: Inputting the image data of the to-be-matched product into the recognition model to obtain the text information of the to-be-matched product; Inputting the image data of the to-be-matched product into the multimodal macro model to obtain appearance description information of the to-be-matched product; The text information of the to-be-matched product and the appearance description information of the to-be-matched product are integrated to obtain the description information of the to-be-matched product.

3. The method according to claim 2, It is characterized in that The step of inputting the image data of the to-be-matched product into a recognition model to obtain text information of the to-be-matched product includes: Input the image data of the product to be matched into the recognition model to obtain output image feature information; The output image feature information is cleaned to obtain text information of the product to be matched.

4. The method according to claim 2, It is characterized in that Before inputting the image data of the to-be-matched product into the multimodal macro model to obtain the appearance description information of the to-be-matched product, the method further includes: Acquire data of historically successfully matched commodities, wherein the data of historically successfully matched commodities include text information, image data of historically to-be-matched commodities, and description information of commodities corresponding to the historically successfully matched waybills; Pre-training the multimodal large model based on the text information and image data of the historical products to be matched; The parameters of the multimodal large model are corrected based on the description information of the goods corresponding to the historical waybills that have been successfully matched.

5. The method according to any one of claims 2 to 4, It is characterized in that The step of fusing the text information of the to-be-matched product and the appearance description information of the to-be-matched product to obtain the description information of the to-be-matched product includes: The text information of the to-be-matched product and the appearance description information of the to-be-matched product are input into a natural language processing model to obtain the description information of the to-be-matched product.

6. The method according to claim 1, It is characterized in that The calculating the similarity based on the description information and the description information of the commodity corresponding to the suspected waybill includes: Performing vectorization processing on the description information and the description information of the commodity corresponding to the suspected waybill respectively; Similarity calculation is performed based on the vectorized processing results.

7. The method according to claim 6, It is characterized in that The determining the matching result of the to-be-matched product based on the result of the similarity calculation includes: Based on the result of the similarity calculation, the commodities corresponding to the suspected waybills are sorted in descending order of similarity to obtain an ordered sequence; A matching result of the to-be-matched commodities is determined based on the ordered sequence.

8. A device for matching goods. It is characterized in that include: The acquisition module is used to obtain the image data and initial waybill data of the product to be matched; An input module, used to input the image data of the to-be-matched product into the model to obtain the description information of the to-be-matched product; A matching module, configured to match the description information with the initial waybill data to obtain a suspected waybill; A calculation module, configured to calculate a similarity between the description information and the description information of the commodity corresponding to the suspected waybill; A determination module is used to determine the matching result of the to-be-matched product based on the result of the similarity calculation.

9. An electronic device, It is characterized in that include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images