Commodity information extraction method and device, storage medium and program product

Abstract

The embodiment of the invention provides a commodity information extraction method and device, a storage medium and a program product. The method comprises the steps of obtaining related images of commodities; identifying first commodity information of the commodity from the related image of the commodity; taking the related image of the commodity and the first commodity information as multi-modal data of the commodity, and generating a target commodity information extraction instruction according to the multi-modal data of the commodity; and calling a preset model according to the target commodity information extraction instruction, and generating structured commodity detail information. The first commodity information is preliminarily extracted from the related image of the commodity, and then the preset model is called based on the first commodity information and the related image of the commodity to extract the target commodity information, so that the accuracy and efficiency of commodity information extraction are improved, the complexity and resource consumption of the preset model are reduced, the manual marking cost is reduced, and the user experience is improved. And the method has a wide application scene, and provides an efficient and reliable solution for various e-commerce platforms and commodity management systems.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer and network communication technology, and in particular, to a commodity information extraction method, device, storage medium, and program product. Background Art

[0002] In scenarios such as e-commerce platforms, it is usually necessary to obtain product details, such as product name, production date, nutritional information, storage conditions, and other information.

[0003] In a related technology, product information is extracted from product packaging images through models such as OCR (Optical Character Recognition), but its accuracy is not high, and models such as OCR require manual annotation, and the cost of manual annotation is high. Summary of the invention

[0004] The embodiments of the present disclosure provide a method, device, storage medium and program product for extracting product information to improve the accuracy of extracting product information.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for extracting commodity information, comprising:

[0006] Get relevant images of the product;

[0007] Identifying first product information of the product from the related image of the product;

[0008] Using the related image of the commodity and the first commodity information as the multimodal data of the commodity, and generating a target commodity information extraction instruction according to the multimodal data of the commodity;

[0009] The preset model is called according to the target product information extraction instruction, the target product information of the product is extracted through the preset model, and structured product detail information is generated according to the target product information.

[0010] In a second aspect, an embodiment of the present disclosure provides a commodity information extraction device, including:

[0011] An acquisition unit, used for acquiring relevant images of the product;

[0012] an identification unit, configured to identify first commodity information of the commodity from the relevant image of the commodity;

[0013] an instruction generating unit, configured to use the related image of the commodity and the first commodity information as the multimodal data of the commodity, and to generate a target commodity information extraction instruction according to the multimodal data of the commodity;

[0014] The information extraction unit is used to call a preset model according to the target product information extraction instruction, extract the target product information of the product through the preset model, and generate structured product details information according to the target product information.

[0015] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0016] The memory stores computer-executable instructions;

[0017] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the commodity information extraction method as described in the first aspect and various possible designs of the first aspect.

[0018] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the commodity information extraction method described in the first aspect and various possible designs of the first aspect is implemented.

[0019] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the commodity information extraction method described in the first aspect and various possible designs of the first aspect.

[0020] The commodity information extraction method, device, storage medium and program product provided by the embodiments of the present disclosure obtain the relevant image of the commodity; identify the first commodity information of the commodity from the relevant image of the commodity; use the relevant image of the commodity and the first commodity information as the multimodal data of the commodity, and generate a target commodity information extraction instruction according to the multimodal data of the commodity; call a preset model according to the target commodity information extraction instruction, extract the target commodity information of the commodity through the preset model, and generate structured commodity details information according to the target commodity information. By first preliminarily extracting the first commodity information from the relevant image of the commodity, and then calling the preset model based on the first commodity information and the relevant image of the commodity to extract the target commodity information, the accuracy and efficiency of commodity information extraction can be improved, and at the same time, the calculation amount of the preset model is reduced with the help of the first commodity information, the complexity and resource consumption of the preset model are reduced, and the cost of manual labeling is also reduced. It has a wide range of applicable scenarios and provides an efficient and reliable solution for various e-commerce platforms and commodity management systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0022] Figure 1 A schematic diagram of a scenario of a method for extracting commodity information provided by an embodiment of the present disclosure;

[0023] Figure 2 A flowchart of a method for extracting commodity information provided by an embodiment of the present disclosure;

[0024] Figure 3 A schematic diagram of processing multimodal data in a commodity information extraction method provided in an embodiment of the present disclosure;

[0025] Figure 4 A structural block diagram of a commodity information extraction device provided in an embodiment of the present disclosure;

[0026] Figure 5 A schematic diagram of the hardware structure of a commodity information extraction device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0028] In scenarios such as e-commerce platforms, it is usually necessary to obtain detailed product information, such as product name, production date, nutritional information table, storage conditions, etc. In a related technology, product information is extracted from product packaging images through models such as OCR, but its accuracy is not high, and models such as OCR require manual annotation, and the cost of manual annotation is high.

[0029] In order to solve the above technical problems, an embodiment of the present disclosure provides a method for extracting product information. On the basis of using models such as OCR to identify product information on related images of products, the product information is used as auxiliary information, and a language model is used to extract product information based on the product information and related images of the products to improve the accuracy of product information extraction.

[0030] Specifically, the method can be implemented by obtaining the relevant images of the product; identifying the first product information of the product from the relevant images of the product; using the relevant images of the product and the first product information as the multimodal data of the product, and generating a target product information extraction instruction according to the multimodal data of the product; calling a preset model according to the target product information extraction instruction, extracting the target product information of the product through the preset model, and generating structured product details information according to the target product information. By first preliminarily extracting the first product information from the relevant images of the product, and then calling the preset model based on the first product information and the relevant images of the product to extract the target product information, the accuracy and efficiency of product information extraction can be improved. At the same time, the calculation amount of the preset model is reduced with the help of the first product information, the complexity and resource consumption of the preset model are reduced, and the cost of manual labeling is also reduced. The method has a wide range of applicable scenarios and provides an efficient and reliable solution for various e-commerce platforms and product management systems.

[0031] The application scenarios of the commodity information extraction method of the disclosed embodiment are as follows: Figure 1 As shown, relevant images of the product can be obtained, and the first product information of the product can be identified from the relevant images of the product. The relevant images of the product and the first product information of the product are used as multimodal data of the product to generate target product information extraction instructions, call the preset model, and generate structured product details information.

[0032] When the first product information of the product is identified from the related images of the product, the table and the non-table can be identified separately according to the product information type, wherein the table detection can be performed to obtain the table information, and the recognition model can be used to perform text recognition on the non-table area to obtain the text information, and the table information and / or the text information is used as the first product information. In addition, before the first product information is identified, it can also be determined whether the related images of the product contain the product information. If the product information is not contained, it returns empty, that is, the related images of the product are filtered out.

[0033] In addition, the second product information of the product can be obtained from the product database, and / or the demand information input by the user can be obtained, and the second product information and / or the demand information can also be added to the multimodal data of the product to generate a target product information extraction instruction.

[0034] The commodity information extraction method disclosed in the present invention will be described in detail below in conjunction with specific embodiments.

[0035] refer to Figure 2 , Figure 2 A flowchart of a method for extracting product information provided by an embodiment of the present disclosure is provided. The method of this embodiment can be applied in a terminal device or a server, and the method for extracting product information includes:

[0036] S201, obtaining relevant images of the product.

[0037] In this embodiment, in order to obtain detailed information of a product, such as product name, production date, nutritional composition table, storage conditions, etc., relevant images of the product may be obtained, and the relevant images of the product include but are not limited to product packaging images, product advertising images, product introduction images, etc. Since the relevant images of the product usually include a lot of product information, the product information may be extracted based on the relevant images of the product. The relevant images of the product may be obtained by any means, such as photographing the packaging of the product through an image acquisition device, or obtaining relevant images of the product from the manufacturer of the product or the user of the product.

[0038] Optionally, since there may be multiple related images of the same product, for example, the packaging of a product has multiple sides, and each side may obtain one or more related images of the product, or there may be related images of products from different sources, and some of the related images of the multiple products may contain product information, and some of the related images of the products may not contain product information. In order to avoid wasting resources by processing related images of products that do not contain product information in subsequent steps, after obtaining related images of multiple products, in this embodiment, it is possible to detect whether the related image of each product in the related images of the multiple products is a related image of a product that contains product information, and filter out related images of products that do not contain product information based on the detection results. Among them, detecting whether the related image of a product is a related image of a product that contains product information, that is, detecting whether there is product information in the related image of a product, can adopt any known method, which is not limited here.

[0039] S202: Identify first product information of the product from related images of the product.

[0040] In this embodiment, preliminary product information recognition can be performed on the relevant images of the product. Any known model for image information recognition can be used, such as an OCR model. These models are relatively mature, consume less resources, and have low costs. However, there are certain recognition errors or there are situations where recognition is impossible. The recognized first product information can be used as a reference or assistance for subsequent steps. By combining the recognition capabilities of models such as OCR with the recognition capabilities of preset models, the advantages of models such as OCR and preset models are fully utilized, which can improve the accuracy and efficiency of extracting product information, and also reduce the size of the preset model.

[0041] In an optional embodiment, considering that there may be a table in the relevant image of the product, and the table is generally in printed fonts, and most of the content is simple such as numbers, the preset model can be used to accurately extract the product information contained in the table, without the need to use OCR or other models for recognition first, thus avoiding waste of resources. Based on the above reasons, when identifying the first product information of the product from the relevant image of the product, it can specifically include:

[0042] Performing table recognition on the image related to the product, if a table is recognized in the image related to the product, obtaining table area coordinates or table area image, and determining the table area coordinates or table area image as table information; and / or

[0043] By using the recognition model, text recognition is performed on the non-table area of ​​the image related to the product to obtain text information;

[0044] The text information and / or the table information is determined as the first product information.

[0045] In this embodiment, table recognition can be performed on the relevant images of the goods. If a table is recognized in the relevant images of the goods, the coordinates of the table area can be obtained as the table information, or the table area image can be further intercepted from the relevant images of the goods according to the table area coordinates, and the table area image can be used as the table information. For non-table areas, OCR and other models are still used to perform product information recognition to obtain text information. Finally, the text information and / or table information can be determined as the first product information, wherein only a table may exist in the relevant images of the goods, and only the table information is obtained as the first product information, or there is no table in the relevant images of the goods, and only the text information is obtained as the first product information, or there may be both a table and text other than the table in the relevant images of the goods, and the text information and the table information can be obtained as the first product information.

[0046] The table recognition of the relevant images of the product can be performed using any known method, such as an image recognition model, etc., which is not limited here.

[0047] In another optional embodiment, the table in the relevant image of the product may not be considered separately, and the relevant image of the product may be subjected to text recognition through a recognition model to obtain text information, and the text information is determined as the first product information. That is, text recognition is performed on all areas in the relevant image of the product through an OCR model, and text recognition is also performed on the table area, and all are included in the text information as the first product information.

[0048] It should be noted that although OCR and other models are also used in this embodiment and certain manual labeling is required, the accuracy of OCR and other models is not highly dependent. Therefore, manual labeling can be reduced, or a general recognition model can be used to reduce the cost of manual labeling.

[0049] S203: Use the relevant image of the product and the first product information as multimodal data of the product, and generate a target product information extraction instruction according to the multimodal data of the product.

[0050] In this embodiment, after obtaining the first product information, a preset model can be called based on the relevant image of the product and the first product information, wherein the preset model, such as a large language model (LLM), usually requires an input instruction (Prompt) in the form of natural language text, and the input instruction is used to describe the task that needs to be performed by the language model to guide the language model to perform the task and generate the required result. In this embodiment, the relevant image of the product and the first product information can be used as multimodal data to generate a target product information extraction instruction (Prompt) as an input of the preset model to instruct the preset model to extract the product information. The multimodal data includes text modal data and image modal data, wherein the text modal data includes the text information in the first product information, and if the table information adopts the table area coordinates, the table area coordinates are also text modal data; and the image modal data includes the relevant image of the product, and if the table information adopts the table area image, the table area image is also image modal data.

[0051] As an example, the target product information extraction instruction Prompt may be:

[0052] “Please extract the target product information based on the product’s related images and the first product information identified from the product’s related images;

[0053] The first product information includes [product name A, production date 2024-01-01, storage condition B, shelf life n months, ...];

[0054] The product's related image is [XX], and the table area image is [YY]. "

[0055] Based on any of the above embodiments, generating a target product information extraction instruction according to the multimodal data of the product in S203 may specifically include:

[0056] Generate a first target product information extraction instruction according to the text modal data in the multimodal data and a preset instruction template;

[0057] The image modality data in the multimodal data is converted into image embedding features that can be recognized by the preset model, and a second target product information extraction instruction is obtained according to the image embedding features.

[0058] In this embodiment, since the preset model, such as the large language model, can only process natural language tasks, it is necessary to convert the image modality data into text representation. Figure 3 As shown, in this embodiment, the text modality data and the image modality data need to be processed separately, wherein the text modality data can directly generate the target product information extraction instruction Prompt as shown in the above example according to the preset instruction template, which is recorded as the first target product information extraction instruction, as a part of the target product information extraction instruction, that is, a token of the preset model; and the image modality data needs to be converted into an image embedding feature that can be recognized by the preset model, that is, a text representation of the image, and then the second target product information extraction instruction is obtained according to the image embedding feature, as another part of the target product information extraction instruction, that is, another token of the preset model. Optionally, the image embedding feature can be used as a separate input of the preset model, or it can also be integrated into the first target product information extraction instruction. For example, in the target product information extraction instruction Prompt shown in the above example, "the relevant image of the product is [XX], and the table area image is [YY]", [XX] and [YY] can be the identifiers (or addresses) of the image embedding features of the relevant image of the product and the table area image, or [XX] and [YY] can also be directly the image embedding features of the relevant image of the product and the table area image.

[0059] Optionally, the image modality data in the multimodal data is converted into image embedding features that can be recognized by the preset model. Specifically, image features can be extracted from the image modality data, for example, by using an encoder to extract image features, and the image features are mapped to the semantic space of the preset model through an image mapping unit to obtain image embedding features that can be recognized by the preset model. Of course, other methods can also be used to convert image modality data into text modality data, which will not be elaborated here.

[0060] S204: calling a preset model according to the target product information extraction instruction, extracting the target product information of the product through the preset model, and generating structured product detail information according to the target product information.

[0061] In this embodiment, the target product information extraction instruction is input into the preset model, and the preset model extracts the target product information according to the input target product information extraction instruction. The preset model includes but is not limited to a large language model, and the preset model can be pre-trained (or fine-tuned) so that it can output the target product information according to the input target product information extraction instruction, wherein the target product information can be predetermined, for example, the preset model is specified to extract which product information during training, or it can be personalized by the user, for example, the target product information required by the user can be added to the target product information extraction instruction. The process of extracting the target product information by the preset model is not limited in this embodiment.

[0062] Furthermore, after the preset model extracts the target product information, it can output the target product information in a specific format, such as a specified JSON format, etc., so as to obtain structured product details information, which includes the above-mentioned target product information. Similarly, the format of the structured product details information can also be predetermined, such as specifying the format during training, or can also be customized by the user, such as adding the target format required by the user to the target product information extraction instruction.

[0063] Based on any of the above embodiments, after obtaining structured product detail information, the product detail information may be stored in a product database for subsequent query of the product detail information; or the product detail information may be sent to a user who requests the product detail information.

[0064] In another embodiment, the product details information may also be rendered to generate a product details page, which may be used to display the product details information and may be used in scenarios such as e-commerce platforms or product management systems.

[0065] The commodity information extraction method provided in this embodiment obtains the relevant image of the commodity; identifies the first commodity information of the commodity from the relevant image of the commodity; uses the relevant image of the commodity and the first commodity information as the multimodal data of the commodity, and generates a target commodity information extraction instruction according to the multimodal data of the commodity; calls a preset model according to the target commodity information extraction instruction, extracts the target commodity information of the commodity through the preset model, and generates structured commodity details information according to the target commodity information. By first preliminarily extracting the first commodity information from the relevant image of the commodity, and then calling the preset model based on the first commodity information and the relevant image of the commodity to extract the target commodity information, the accuracy and efficiency of commodity information extraction can be improved, and at the same time, the calculation amount of the preset model is reduced with the help of the first commodity information, and the complexity and resource consumption of the preset model are reduced. In this embodiment, the cost of manual labeling is also reduced, and it has a wide range of applicable scenarios, providing an efficient and reliable solution for various e-commerce platforms and commodity management systems.

[0066] On the basis of any of the above embodiments, some product information of the product may also exist in the product database, which can also be used as auxiliary data or reference data for extracting product information by the preset model to further improve the accuracy of product information extraction. Specifically, in this embodiment, the second product information of the product can be obtained from the product database, and then when generating the target product information extraction instruction, the second product information is also added to the multimodal data of the product, and then the target product information extraction instruction is generated based on the multimodal data of the product. As an example, the target product information extraction instruction Prompt can be:

[0067] “Please extract the target product information based on the relevant image of the product, the first product information identified from the relevant image of the product, and the second product information obtained from the product database;

[0068] The first product information includes [product name A, production date 2024-01-01, storage conditions B, shelf life n months, ...]; the second product information includes [product name A, product description C, ...];

[0069] The product's related image is [XX], and the table area image is [YY]. "

[0070] Optionally, when obtaining the second product information of the product from the product database, the product database may be searched according to one or more items of the first product information. Optionally, the product information stored in the product database may be vectorized in advance, and then vector search may be used during the search, and the obtained second product information may also be vectorized second product information.

[0071] On the basis of any of the above embodiments, the user can set the required product information type for the target product information, and can also set the structured product details information format. Specifically, the user's input requirement information can be obtained, and the requirement information includes the required product information type and / or product details information format; then when generating the target product information extraction instruction, the requirement information is also added to the multimodal data of the product, and then the target product information extraction instruction is generated according to the multimodal data of the product, so that the preset model can extract the product information required by the user and / or output it in the format required by the user, thereby improving the freedom and flexibility of product information extraction. As an example, the target product information extraction instruction Prompt can be:

[0072] “Please extract target product information M1, target product information M2, and target product information M3 based on the product related images, the first product information identified from the product related images, and the second product information obtained from the product database, and output them in product detail information format G;

[0073] The first product information includes [product name A, production date 2024-01-01, storage conditions B, shelf life n months, ...]; the second product information includes [product name A, product description C, ...];

[0074] The product's related image is [XX], and the table area image is [YY]. "

[0075] Corresponding to the commodity information extraction method of the above embodiment, Figure 4 This is a structural block diagram of a commodity information extraction device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 4 The commodity information extraction device 400 includes: an acquisition unit 401, an identification unit 402, an instruction generation unit 403, and an information extraction unit 404.

[0076] The acquisition unit is used to acquire the relevant images of the product;

[0077] an identification unit, configured to identify first commodity information of the commodity from the relevant image of the commodity;

[0078] an instruction generating unit, configured to use the related image of the commodity and the first commodity information as the multimodal data of the commodity, and to generate a target commodity information extraction instruction according to the multimodal data of the commodity;

[0079] The information extraction unit is used to call a preset model according to the target product information extraction instruction, extract the target product information of the product through the preset model, and generate structured product details information according to the target product information.

[0080] In one or more embodiments of the present disclosure, when the recognition unit recognizes the first commodity information of the commodity from the related image of the commodity, it is used to:

[0081] Performing table recognition on the image related to the product, if a table is recognized in the image related to the product, obtaining table area coordinates or table area image, and determining the table area coordinates or table area image as table information; and / or

[0082] By using the recognition model, text recognition is performed on the non-table area of ​​the image related to the product to obtain text information;

[0083] The text information and / or the table information is determined as the first product information.

[0084] In one or more embodiments of the present disclosure, when the recognition unit recognizes the first commodity information of the commodity from the related image of the commodity, it is used to:

[0085] Through the recognition model, text recognition is performed on the relevant image of the product to obtain text information, and the text information is determined as the first product information.

[0086] In one or more embodiments of the present disclosure, the acquisition unit is further configured to:

[0087] Acquire second commodity information of the commodity from a commodity database;

[0088] Accordingly, when the instruction generation unit generates the target product information extraction instruction according to the multimodal data of the product, it is used to:

[0089] The second product information is added to the multimodal data of the product, and a target product information extraction instruction is generated according to the multimodal data of the product.

[0090] In one or more embodiments of the present disclosure, the acquisition unit is further configured to:

[0091] Acquire demand information input by a user, wherein the demand information includes a required product information type and / or product detail information format;

[0092] Accordingly, when the instruction generating unit uses the related image of the commodity and the first commodity information as the multimodal data of the commodity and generates the target commodity information extraction instruction according to the multimodal data of the commodity, it is used to:

[0093] The demand information is added to the multimodal data of the commodity, and a target commodity information extraction instruction is generated according to the multimodal data of the commodity.

[0094] In one or more embodiments of the present disclosure, when the instruction generation unit generates the target product information extraction instruction according to the multimodal data of the product, it is used to:

[0095] Generate a first target product information extraction instruction according to the text modal data in the multimodal data and a preset instruction template;

[0096] The image modality data in the multimodal data is converted into image embedding features that can be recognized by the preset model, and a second target product information extraction instruction is obtained according to the image embedding features.

[0097] In one or more embodiments of the present disclosure, when converting the image modality data in the multimodal data into image embedding features that can be recognized by the preset model, the instruction generation unit is used to:

[0098] Image features are extracted from the image modality data in the multimodal data, and the image features are mapped to the semantic space of the preset model to obtain image embedding features that can be recognized by the preset model.

[0099] In one or more embodiments of the present disclosure, there are multiple related images of the commodity; before identifying the first commodity information of the commodity from the related images of the commodity, the acquiring unit is further configured to:

[0100] detecting whether each product-related image among a plurality of product-related images of the product is a product-related image containing product information;

[0101] According to the detection results, the relevant images of the products that do not contain product information are filtered out.

[0102] In one or more embodiments of the present disclosure, after generating structured product details information according to the target product information, the information extraction unit is further configured to:

[0103] The product details information is rendered to generate a product details page.

[0104] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0105] In order to implement the above embodiment, the embodiment of the present disclosure also provides an electronic device.

[0106] refer to Figure 5, which shows a schematic diagram of the structure of an electronic device 500 suitable for implementing the embodiment of the present disclosure, and the electronic device 500 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0107] like Figure 5 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 to a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0108] Typically, the following devices may be connected to the I / O interface 505: input devices 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 508 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 509. The communication devices 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0109] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0110] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0111] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0112] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0113] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0114] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0115] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0116] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0117] In a first aspect, according to one or more embodiments of the present disclosure, a method for extracting commodity information is provided, comprising:

[0118] Get relevant images of the product;

[0119] Identifying first product information of the product from the related image of the product;

[0120] Using the related image of the commodity and the first commodity information as the multimodal data of the commodity, and generating a target commodity information extraction instruction according to the multimodal data of the commodity;

[0121] The preset model is called according to the target product information extraction instruction, the target product information of the product is extracted through the preset model, and structured product detail information is generated according to the target product information.

[0122] According to one or more embodiments of the present disclosure, the step of identifying first product information of the product from the related image of the product includes:

[0123] Performing table recognition on the image related to the product, if a table is recognized in the image related to the product, obtaining table area coordinates or table area image, and determining the table area coordinates or table area image as table information; and / or

[0124] By using the recognition model, text recognition is performed on the non-table area of ​​the image related to the product to obtain text information;

[0125] The text information and / or the table information is determined as the first product information.

[0126] According to one or more embodiments of the present disclosure, the step of identifying first product information of the product from the related image of the product includes:

[0127] Through the recognition model, text recognition is performed on the relevant image of the product to obtain text information, and the text information is determined as the first product information.

[0128] According to one or more embodiments of the present disclosure, the method further includes:

[0129] Acquire second commodity information of the commodity from a commodity database;

[0130] Accordingly, generating a target product information extraction instruction according to the multimodal data of the product includes:

[0131] The second product information is added to the multimodal data of the product, and a target product information extraction instruction is generated according to the multimodal data of the product.

[0132] According to one or more embodiments of the present disclosure, the method further includes:

[0133] Acquire demand information input by a user, wherein the demand information includes a required product information type and / or product detail information format;

[0134] Correspondingly, taking the related image of the commodity and the first commodity information as the multimodal data of the commodity and generating a target commodity information extraction instruction according to the multimodal data of the commodity includes:

[0135] The demand information is added to the multimodal data of the commodity, and a target commodity information extraction instruction is generated according to the multimodal data of the commodity.

[0136] According to one or more embodiments of the present disclosure, the step of generating a target product information extraction instruction according to the multimodal data of the product includes:

[0137] Generate a first target product information extraction instruction according to the text modal data in the multimodal data and a preset instruction template;

[0138] The image modality data in the multimodal data is converted into image embedding features that can be recognized by the preset model, and a second target product information extraction instruction is obtained according to the image embedding features.

[0139] According to one or more embodiments of the present disclosure, converting the image modality data in the multimodal data into image embedding features that can be recognized by the preset model includes:

[0140] Image features are extracted from the image modality data in the multimodal data, and the image features are mapped to the semantic space of the preset model to obtain image embedding features that can be recognized by the preset model.

[0141] According to one or more embodiments of the present disclosure, there are multiple related images of the commodity; before identifying the first commodity information of the commodity from the related images of the commodity, the method further includes:

[0142] detecting whether each product-related image among a plurality of product-related images of the product is a product-related image containing product information;

[0143] According to the detection results, the relevant images of the products that do not contain product information are filtered out.

[0144] According to one or more embodiments of the present disclosure, after generating structured product details information according to the target product information, the method further includes:

[0145] The product details information is rendered to generate a product details page.

[0146] In a second aspect, according to one or more embodiments of the present disclosure, a commodity information extraction device is provided, including:

[0147] An acquisition unit, used for acquiring relevant images of the product;

[0148] an identification unit, configured to identify first commodity information of the commodity from the relevant image of the commodity;

[0149] an instruction generating unit, configured to use the related image of the commodity and the first commodity information as the multimodal data of the commodity, and to generate a target commodity information extraction instruction according to the multimodal data of the commodity;

[0150] The information extraction unit is used to call a preset model according to the target product information extraction instruction, extract the target product information of the product through the preset model, and generate structured product details information according to the target product information.

[0151] According to one or more embodiments of the present disclosure, when the recognition unit recognizes the first commodity information of the commodity from the related image of the commodity, it is used to:

[0152] Performing table recognition on the image related to the product, if a table is recognized in the image related to the product, obtaining table area coordinates or table area image, and determining the table area coordinates or table area image as table information; and / or

[0153] By using the recognition model, text recognition is performed on the non-table area of ​​the image related to the product to obtain text information;

[0154] The text information and / or the table information is determined as the first product information.

[0155] According to one or more embodiments of the present disclosure, when the recognition unit recognizes the first commodity information of the commodity from the related image of the commodity, it is used to:

[0156] Through the recognition model, text recognition is performed on the relevant image of the product to obtain text information, and the text information is determined as the first product information.

[0157] According to one or more embodiments of the present disclosure, the acquisition unit is further configured to:

[0158] Acquire second commodity information of the commodity from a commodity database;

[0159] Accordingly, when the instruction generation unit generates the target product information extraction instruction according to the multimodal data of the product, it is used to:

[0160] The second product information is added to the multimodal data of the product, and a target product information extraction instruction is generated according to the multimodal data of the product.

[0161] According to one or more embodiments of the present disclosure, the acquisition unit is further configured to:

[0162] Acquire demand information input by a user, wherein the demand information includes a required product information type and / or product detail information format;

[0163] Accordingly, when the instruction generating unit uses the related image of the commodity and the first commodity information as the multimodal data of the commodity and generates the target commodity information extraction instruction according to the multimodal data of the commodity, it is used to:

[0164] The demand information is added to the multimodal data of the commodity, and a target commodity information extraction instruction is generated according to the multimodal data of the commodity.

[0165] According to one or more embodiments of the present disclosure, when the instruction generation unit generates the target product information extraction instruction according to the multimodal data of the product, it is used to:

[0166] Generate a first target product information extraction instruction according to the text modal data in the multimodal data and a preset instruction template;

[0167] The image modality data in the multimodal data is converted into image embedding features that can be recognized by the preset model, and a second target product information extraction instruction is obtained according to the image embedding features.

[0168] According to one or more embodiments of the present disclosure, when converting the image modality data in the multimodal data into image embedding features that can be recognized by the preset model, the instruction generation unit is used to:

[0169] Image features are extracted from the image modality data in the multimodal data, and the image features are mapped to the semantic space of the preset model to obtain image embedding features that can be recognized by the preset model.

[0170] According to one or more embodiments of the present disclosure, there are multiple related images of the commodity; before identifying the first commodity information of the commodity from the related images of the commodity, the acquisition unit is further used to:

[0171] detecting whether each product-related image among a plurality of product-related images of the product is a product-related image containing product information;

[0172] According to the detection results, the relevant images of the products that do not contain product information are filtered out.

[0173] According to one or more embodiments of the present disclosure, after generating structured product details information according to the target product information, the information extraction unit is further configured to:

[0174] The product details information is rendered to generate a product details page.

[0175] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0176] The memory stores computer-executable instructions;

[0177] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the commodity information extraction method as described in the first aspect and various possible designs of the first aspect.

[0178] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the commodity information extraction method described in the first aspect and various possible designs of the first aspect is implemented.

[0179] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the method for extracting product information as described in the first aspect and various possible designs of the first aspect is implemented.

[0180] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0181] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0182] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for extracting commodity information, characterized in that: include: Get relevant images of the product; Identifying first product information of the product from the related image of the product; Using the relevant image of the commodity and the first commodity information as the multimodal data of the commodity, and generating a target commodity information extraction instruction according to the multimodal data of the commodity; The preset model is called according to the target product information extraction instruction, the target product information of the product is extracted through the preset model, and structured product detail information is generated according to the target product information.

2. The method according to claim 1, characterized in that The step of identifying the first commodity information of the commodity from the related image of the commodity includes: Performing table recognition on the image related to the product, if a table is recognized in the image related to the product, obtaining table area coordinates or table area image, and determining the table area coordinates or table area image as table information; and / or By using the recognition model, text recognition is performed on the non-table area of ​​the image related to the product to obtain text information; The text information and / or the table information is determined as the first product information.

3. The method according to claim 1, characterized in that The step of identifying the first commodity information of the commodity from the related image of the commodity includes: Through the recognition model, text recognition is performed on the relevant image of the product to obtain text information, and the text information is determined as the first product information.

4. The method according to claim 1, characterized in that: The method further comprises: Acquire second commodity information of the commodity from a commodity database; Accordingly, generating a target product information extraction instruction according to the multimodal data of the product includes: The second product information is added to the multimodal data of the product, and a target product information extraction instruction is generated according to the multimodal data of the product.

5. The method according to claim 1, characterized in that The method further comprises: Acquire demand information input by a user, wherein the demand information includes a required product information type and / or product detail information format; Correspondingly, taking the related image of the commodity and the first commodity information as the multimodal data of the commodity and generating a target commodity information extraction instruction according to the multimodal data of the commodity includes: The demand information is added to the multimodal data of the commodity, and a target commodity information extraction instruction is generated according to the multimodal data of the commodity.

6. The method according to any one of claims 1 to 5, characterized in that: The step of generating a target product information extraction instruction according to the multimodal data of the product includes: Generate a first target product information extraction instruction according to the text modal data in the multimodal data and a preset instruction template; The image modality data in the multimodal data is converted into image embedding features that can be recognized by the preset model, and a second target product information extraction instruction is obtained according to the image embedding features.

7. The method according to claim 6, characterized in that The converting the image modality data in the multimodal data into image embedding features that can be recognized by the preset model includes: Image features are extracted from the image modality data in the multimodal data, and the image features are mapped to the semantic space of the preset model to obtain image embedding features that can be recognized by the preset model.

8. The method according to claim 1, characterized in that There are multiple related images of the commodity; before identifying the first commodity information of the commodity from the related images of the commodity, the method further includes: detecting whether each product-related image among a plurality of product-related images of the product is a product-related image containing product information; According to the detection results, the relevant images of the products that do not contain product information are filtered out.

9. The method according to claim 1, characterized in that: After generating structured product details information according to the target product information, the method further includes: The product details information is rendered to generate a product details page.

10. A commodity information extraction device, characterized in that: include: An acquisition unit, used for acquiring relevant images of the product; an identification unit, configured to identify first commodity information of the commodity from the relevant image of the commodity; an instruction generating unit, configured to use the related image of the commodity and the first commodity information as the multimodal data of the commodity, and to generate a target commodity information extraction instruction according to the multimodal data of the commodity; The information extraction unit is used to call a preset model according to the target product information extraction instruction, extract the target product information of the product through the preset model, and generate structured product details information according to the target product information.

11. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images