A product information extraction method and device

Abstract

The embodiment of the application relates to the computer technical field, and discloses a product information extraction method and device, the method comprises the following steps: obtaining target text information in a target image and context relationships of each character in the target text information; calling a trained named entity recognition model to process the target text information and the context relationships of each character, and obtaining entity labels of each character; determining at least one entity label group in the entity labels of each character, and the entity labels contained in each entity label group are of the same label type; and combining characters corresponding to each entity label in the same entity label group to obtain product information. According to the embodiment of the application, the product information in the image can be accurately extracted through big data training, so that the input efficiency of the product information is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and apparatus for extracting product information. Background Technology

[0002] When merchants begin using new systems such as ordering, review, and promotion systems, the diverse styles, designs, and product information of their original offline product catalogs often lead to errors when directly extracting product information from images containing catalog data using text recognition. Furthermore, manually entering product information is extremely time-consuming and labor-intensive, especially with a large number of products. Therefore, accurately extracting product information from images to improve the efficiency of product information entry is a pressing technical problem that needs to be solved. Summary of the Invention

[0003] This application provides a product information extraction method and apparatus, which can accurately extract product information from images, thereby improving the efficiency of product information input.

[0004] On the one hand, embodiments of this application provide a product information extraction method, including:

[0005] Obtain target text information from the target image, and the contextual relationship of each character in the target text information;

[0006] The trained named entity recognition model is invoked to process the target text information and the contextual relationship of each character to obtain the entity label of each character;

[0007] Among the entity tags of each character, at least one entity tag group is determined, and the entity tags contained in each entity tag group have the same tag type.

[0008] The product information is obtained by combining the characters corresponding to the various entity tags in the same entity tag group.

[0009] On one hand, embodiments of this application provide a product information extraction device, which includes an acquisition unit and a processing unit, wherein:

[0010] The acquisition unit is used to acquire target text information in the target image, as well as the contextual relationship of each character in the target text information;

[0011] The processing unit is also used to call the trained named entity recognition model to process the target text information and the contextual relationship of each character to obtain the entity label of each character;

[0012] The processing unit is further configured to determine at least one entity tag group in the entity tags of each character, wherein the entity tags contained in each entity tag group have the same tag type.

[0013] The processing unit is also used to combine the characters corresponding to each entity label in the same entity label group to obtain product information.

[0014] On one hand, embodiments of this application provide an electronic device, which includes an input interface and an output interface, and further includes:

[0015] A processor, adapted to implement one or more instructions; and,

[0016] A computer storage medium storing one or more instructions, the one or more instructions being adapted to be loaded by the processor and executed by the above-described text data integrity detection method.

[0017] On one hand, embodiments of this application provide a computer storage medium storing computer program instructions, which, when executed by a processor, are used to perform the aforementioned product information extraction method.

[0018] On one hand, embodiments of this application provide a computer program product or computer program, the computer program product or computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium; a processor of an electronic device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, and when the computer instructions are executed by the processor, they are used to perform the above-described product information extraction method.

[0019] In this embodiment, the target text information in the target image and the contextual relationships of each character in the target text information are first obtained; then, a trained named entity recognition model is invoked to process the target text information and the contextual relationships of each character to obtain the entity labels of each character; then, at least one entity label group is determined from the entity labels of each character, wherein the entity labels contained in each entity label group have the same label type; finally, the characters corresponding to each entity label in the same entity label group are combined to obtain product information. This embodiment, by invoking a trained named entity recognition model to annotate each character in the target text information and determine the entity labels of each character, and then combining the characters corresponding to each entity label belonging to the same entity label group, obtains product information. Because the target text information is processed character by character, and the corresponding characters are combined to obtain product information, it avoids the problems of inaccurate recognition caused by text content adhesion and the diverse styles and designs of product information in the target image when directly recognizing product information in the target image through text recognition. This allows for accurate extraction of product information from the image, thereby improving the efficiency of the product information entry system. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the structure of a product information extraction system provided in an embodiment of this application;

[0022] Figure 2 This is a flowchart illustrating a product information extraction method provided in an embodiment of this application;

[0023] Figure 3 This is a schematic diagram illustrating an embodiment of obtaining target text information provided in this application;

[0024] Figure 4 This is a schematic diagram illustrating another method for obtaining target text information provided in an embodiment of this application;

[0025] Figure 5 This is a schematic diagram illustrating another method of obtaining target text information provided in an embodiment of this application;

[0026] Figure 6 This is a schematic diagram illustrating the determination of an entity label group according to an embodiment of this application;

[0027] Figure 7 This is a flowchart illustrating another product information extraction method provided in an embodiment of this application;

[0028] Figure 8 This is a schematic diagram of an encoding process provided in an embodiment of this application;

[0029] Figure 9 This is a schematic diagram of a Span provided in an embodiment of this application;

[0030] Figure 10 This is a schematic diagram illustrating a process for extracting product information provided in an embodiment of this application;

[0031] Figure 11 This is a flowchart illustrating another product information extraction method provided in the embodiments of this application;

[0032] Figure 12 This is a schematic diagram of the structure of a product information extraction device provided in an embodiment of this application;

[0033] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0035] To accurately extract product information from images and improve the efficiency of product information input, this application provides a product information extraction scheme. First, the target text information and the contextual relationships of each character in the target text information are obtained from the target image. Then, a trained named entity recognition model is invoked to process the target text information and the contextual relationships of each character to obtain entity labels for each character. Next, at least one entity label group is determined from the entity labels of each character, and each entity label group contains entity labels of the same label type. Finally, the characters corresponding to the entity labels in the same entity label group are combined to obtain the product information. The target image refers to an image containing product information. The target image can be an image acquired by an electronic device in response to a user's shooting operation, or an image selected from multiple images in an image database in response to a user's selection operation. For example, the target image can be an image of a menu, an image of a gym's price list, etc.

[0036] In one embodiment, the above product information extraction scheme can be executed by a terminal device, which may include any one or more of smartphones, tablets, laptops, desktop computers, smart in-vehicle devices, and smart wearable devices. The terminal device first acquires a target image, then acquires target text information from the target image, and the contextual relationships of each character in the target text information; then, the terminal device calls a trained named entity recognition model to process the target text information and the contextual relationships of each character to obtain entity labels for each character; then, the terminal device determines at least one entity label group from the entity labels of each character, wherein the entity labels contained in each entity label group have the same label type; finally, the terminal device combines the characters corresponding to each entity label in the same entity label group to obtain product information. Optionally, the above product information extraction scheme can be executed by a server, where the server can be an independent server, a cloud server, a server cluster, or a distributed system, etc., and is not limited thereto.

[0037] In another embodiment, the above-described product information extraction scheme can run in a product information extraction system. The product information extraction system may include a terminal device and a server. Specifically, the terminal device can respond to the user's upload operation by uploading the target image to the server. The server first obtains the target text information in the target image and the contextual relationship of each character in the target text information. Then, it calls the trained named entity recognition model to process the target text information and the contextual relationship of each character to obtain the entity label of each character. Then, the server determines at least one entity label group in the entity labels of each character, wherein the entity labels contained in each entity label group have the same label type. Finally, the server combines the characters corresponding to each entity label in the same entity label group to obtain the product information and sends the product information to the terminal device.

[0038] The above product information extraction system can be found in [reference]. Figure 1 As shown, Figure 1The product information extraction system shown may include a terminal device 101 and a server 102. The terminal device 101 may include any one or more of smartphones, tablets, laptops, desktop computers, smart in-vehicle devices, and smart wearable devices. The server 102 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal device 101 and the server 102 may communicate directly or indirectly via wired or wireless communication, which is not limited herein.

[0039] Based on the above product information extraction scheme, this application provides a product information extraction method. See also... Figure 2 This is a flowchart illustrating a product information extraction method provided in an embodiment of this application. Figure 2 The product information extraction method shown can be executed by a server or terminal device. Figure 2 The product information extraction method shown may include the following steps:

[0040] S201, Obtain target text information in the target image, and the contextual relationship of each character in the target text information.

[0041] In this embodiment, the target text information refers to the text information in the target image used to indicate product content. For example, if the target image is an image containing a Chinese restaurant menu, then the target text information could be the dish names, prices, specifications, currency, etc., from the menu, such as "Braised Eggplant 10 yuan / serving, Steamed Crucian Carp 32 yuan / plate." However, text information such as the restaurant's history in the menu is not considered target text information. Furthermore, the contextual relationship of each character in the target text information refers to the position of the characters within the target text information, such as which characters precede character A and which characters follow character A. The contextual relationship of each character in the target text information can be obtained through semantic analysis of the target text information. Optionally, it can also be obtained by sorting the characters in the target image according to their reading order based on their position information within the target text information; this is not limited here.

[0042] In one embodiment, the method for obtaining the target text information may be: 1) performing text detection processing on the target image to obtain at least one text region and the position information of each text region in the target image; 2) sorting the at least one text region based on the position information of each text region in the target image to obtain sorted text regions; 3) concatenating the characters contained in each text region in the sorted text regions to obtain the target text information.

[0043] Specifically, by sorting at least one text region, the character order of the subsequently concatenated target text information can be kept basically consistent with the character order in the target image, and the semantics of the target text information can be kept basically consistent with the semantics in the target image. This helps the subsequently trained named entity recognition model to more accurately label the entity labels of each character.

[0044] Optionally, text detection processing of the target image can be performed using Optical Character Recognition (OCR) technology to obtain at least one text region. Alternatively, text detection processing of the target image can be performed using a trained text detection model to obtain at least one text region. For example, the text detection model can be a deep learning model such as DBNet (a differentiable binarization segmentation network for text detection), CTPN (a network for text detection that includes convolutional neural networks and recurrent neural networks), or SegLink (a convolutional neural network capable of detecting text with rotation angles). Different models can be flexibly selected based on different needs, and are not limited here. The process of training the text detection model is a technique commonly used by those skilled in the art and will not be elaborated upon here.

[0045] In one embodiment, the location information may include the x-coordinate and y-coordinate of the center point of each text region. The method for sorting at least one text region based on its location information in the target image to obtain the sorted text regions may be as follows: First, sort the target text regions in ascending order of the y-coordinate of their center points; if multiple text regions have the same y-coordinate, or if the y-coordinates of their center points differ very little, then it can be determined that the multiple text regions are in the same row. Then, sort the target text regions among the multiple text regions in ascending order of the x-coordinate of their center points, finally obtaining the sorted target text regions.

[0046] For example, please see the appendix. Figure 3The diagram illustrates a method for acquiring target text information. The target image contains six text regions: the center point of text region A is (200, 200); the center point of text region B is (200, 305); the center point of text region C is (198, 399); the center point of text region D is (400, 210); the center point of text region E is (406, 295); and the center point of text region F is (405, 390).

[0047] Then, text regions with a difference of less than 10 pixels in the vertical coordinate of their center points are defined as text regions with the same vertical coordinate. Therefore, it can be determined that text regions A and D have the same vertical coordinate, text regions B and E have the same vertical coordinate, and text regions C and F have the same vertical coordinate. Next, the horizontal coordinates of the center points of text regions A and D, text regions B and E, and text regions C and F are compared. The resulting sorted target text regions are: Text Region A, Text Region D, Text Region B, Text Region E, Text Region C, and Text Region F. Finally, the characters contained in each of the sorted text regions are concatenated to obtain the target text information 302: "Braised Eggplant 10.00 yuan / serving, Steamed Crucian Carp 32.00 yuan / jin, Minced Meat and Vermicelli 20.00 yuan / serving".

[0048] In one embodiment, the positional information of each text region in the target image includes the horizontal and vertical coordinates of each text region in the target image; therefore, based on the positional information of each text region in the target image, sorting at least one text region to obtain the sorted text regions can be done in the following way:

[0049] 1) Based on the ordinate of each text region, determine at least one text region queue. The ordinate of the text region in the target text region queue of the at least one text region queue is less than the ordinate of the text region in the next text region queue of the target text region queue. The target text region queue is any text region queue in the at least one text region queue. The ordinates of each text region in the same text region queue are matched.

[0050] Specifically, the matching of the vertical coordinates of the text regions in the same text region queue can be that the vertical coordinates of the text regions are the same, or the difference between the vertical coordinates of the text regions is within a first preset threshold, which is not limited here. The first preset threshold can be a pixel value such as 20 pixels or 10 pixels, or a length unit such as 10 millimeters or 1 centimeter, which is not limited here.

[0051] Optionally, the method for determining at least one text region queue based on the ordinates of each text region can be as follows: based on the ordinates of each text region, text regions with the same ordinate or whose ordinate differences are within a first preset threshold are identified as the same text region queue, thus obtaining at least one text region queue. Then, the text region queues are arranged according to the ordinates of the text regions in each text region queue, or the average ordinate of the text regions in each text region queue, such that the ordinate of the text region in the target text region queue is less than the ordinate of the text region in the next text region queue of the target text region queue.

[0052] Optionally, at least one text region queue can be determined based on the x-coordinate and y-coordinate of each text region: First, based on the x-coordinate of each text region, the text region with the largest or smallest x-coordinate in the target image is determined as the first text region. If the difference between the x-coordinate of text region M and the x-coordinate of the first text region is less than a second preset threshold, then text region M can also be determined as the first text region. Then, among the text regions other than the first text region, the text regions with the same y-coordinate as the first text region or whose y-coordinate difference is within a preset threshold are determined as the second text regions. Finally, the first and second text regions are determined as the same text region queue. The second preset threshold can be a pixel value such as 20 pixels or 10 pixels, or a length unit such as 10 millimeters or 1 centimeter; it is not limited here.

[0053] 2) Traverse at least one text region queue. Based on the x-coordinate of each text region in the currently traversed text region queue, sort the text regions in the currently traversed text region queue to obtain the sorted text regions. Specifically, sort the text regions in the currently traversed text region queue according to their x-coordinates in ascending order.

[0054] For example, please see the appendix. Figure 4This diagram illustrates another method for acquiring target text information. After performing text detection processing on the target image 401, 12 text regions, namely text regions A to L, are obtained. With a first preset threshold of 10 pixels, text regions A, B, C, and D are identified as belonging to the same text region queue, text regions E, F, G, and H are identified as belonging to the same text region queue, and text regions I, J, K, and L are identified as belonging to the same text region queue. Furthermore, based on the ordinates of text regions A, E, and I, the text region queue containing text region A is identified as the first text region queue, the text region queue containing text region E is identified as the second text region queue, and the text region queue containing text region I is identified as the third text region queue.

[0055] Then, the three text region queues are traversed in the order from the first to the third text region queue. When traversing the first text region queue, based on the ascending order of the horizontal coordinates, the text region order of the first text region queue is determined to be text region A, text region B, text region C, and text region D. When traversing the second text region queue, based on the ascending order of the horizontal coordinates, the text region order of the second text region queue is determined to be text region E, text region F, text region G, and text region H. When traversing the third text region queue, based on the ascending order of the horizontal coordinates, the text region order of the third text region queue is determined to be text region I, text region J, text region K, and text region L. Therefore, the final sorted text regions are: text region A, text region B, text region C, text region D, text region E, text region F, text region G, text region H, text region I, text region J, text region K, and text region L. Finally, the characters contained in each text area in the sorted text area are concatenated to obtain the target text information 402: "Braised eggplant 10.00 yuan / serving, cold donkey meat 40.00 yuan / plate, cold cucumber salad 8.00 yuan / serving, steamed crucian carp 32.00 yuan / jin, stir-fried seasonal vegetables 12.00 yuan / serving, minced meat and vermicelli 20.00 yuan / serving."

[0056] In one embodiment, the method of sorting at least one text region based on the position information of each text region in the target image to obtain the sorted text region can be as follows: calling a word order correction model, performing word order correction processing on at least one text region based on the position information of each text region, obtaining a processed text region sequence, wherein the word order between characters in any text region of the processed text region sequence and characters in the previous text region of any text region, as well as characters in the next text region of any text region, satisfies a preset word order rule; and the processed text region sequence is used as the sorted text region.

[0057] Specifically, the preset word order rule can be a word order rule set manually based on the word order of characters in the product information, or a word order rule summarized by the system based on the word order of characters in the product information; it is not limited here. For example, when the target image is an image containing a menu, the preset word order rule can be set to have the dish name, price, and specifications appear alternately.

[0058] Furthermore, the word order correction model can be obtained by training a machine learning model such as a recurrent neural network (RNN) with a large amount of product-related text information. The process of calling the word order correction model to perform word order correction on at least one text region based on the positional information of each text region, resulting in a processed text region sequence, can be specifically as follows: First, the word order correction model concatenates the characters in each text region based on the positional information of each text region to obtain concatenated text information. Then, the word order correction model adjusts the word order of the characters in the concatenated text information to obtain adjusted concatenated text information that meets a preset word order rule. Finally, at least one text region is sorted according to the characters in the adjusted concatenated text information to obtain a processed text region sequence. Finally, the processed text region sequence is used as the sorted text region.

[0059] For example, please see the appendix. Figure 5 This illustrates another method for obtaining target text information. After performing text detection processing on target image 501, 11 text regions are obtained. Then, the word order correction model is called to concatenate the characters in each text region based on the positional information, resulting in the concatenated text information 502: "Pizza Super Supreme Pizza Durian Pizza Steak / Pasta Orleans Roast Meat Pizza Australian Sirloin Steak Premium Steak Classic Meat Sauce Pasta Spanish Flavor Pasta Snacks Delicious Snack Platter". Since the final menu is categorized by dish type and name, the word order correction model will follow the order of the dishes. The order of the concatenated text information 502 is adjusted based on the category and the order of the dish names belonging to that category, resulting in the adjusted concatenated text information 503: "Super Supreme Pizza, Durian Pizza, Orleans Roast Meat Pizza, Steak / Pasta, Australian Sirloin Steak, Premium Steak, Classic Meat Sauce Pasta, Spanish Flavor Pasta Snacks, Delicious Snack Platter". Thus, the sorted text regions can be determined as text region a, text region b, text region c, text region e, text region d, text region f, text region g, text region i, text region j, and text region k.

[0060] Optionally, to avoid situations where the text region is tilted and difficult to sort during text detection processing of the captured target image due to the page being tilted during the process of shooting menus, item price lists, etc., the target image can be angle-corrected before text detection processing, so that the menus, item price lists, etc. in the target image are adjusted to be upright, that is, the page is in a non-tilted state.

[0061] Optionally, in some real-world scenarios, the text on pages such as menus and item price lists may be arranged vertically. Therefore, during the text detection processing of the target image, the vertically arranged text can be converted into horizontally arranged text. Then, referring to the above embodiments, the text regions can be sorted based on the horizontal and vertical coordinates of each text region and by calling the trained word order correction model.

[0062] In addition, the text arrangement can be determined during the text detection process of the target image. If the arrangement is horizontal, the text regions can be sorted by referring to the method based on the horizontal and vertical coordinates of each text region mentioned in the above embodiments, as well as by calling the trained word order correction model.

[0063] If the arrangement is vertical, then based on the position information of each text region in the target image, at least one text region is sorted to obtain the sorted text regions. The sorted text regions can be obtained by: determining at least one text region queue based on the horizontal coordinate of each text region, wherein the horizontal coordinate of the text region in the target text region queue is less than the horizontal coordinate of the text region in the next text region queue of the target text region queue, and the target text region queue is any text region queue in the at least one text region queue, and the horizontal coordinates of each text region in the same text region queue are matched; traversing at least one text region queue, and sorting each text region in the currently traversed text region queue based on the vertical coordinate of each text region in the currently traversed text region queue to obtain the sorted text regions.

[0064] Meanwhile, the method of sorting at least one text region based on the position information of each text region in the target image to obtain the sorted text region can also be: calling a word order correction model, performing word order correction processing on at least one text region based on the position information of each text region, obtaining a processed text region sequence, wherein the word order between the characters in any text region of the processed text region sequence and the characters in the previous text region of any text region, as well as the characters in the next text region of any text region, satisfies a preset word order rule; and the processed text region sequence is used as the sorted text region.

[0065] The specific implementation method for sorting text regions when arranging them vertically can be found in the above embodiments, which uses the horizontal and vertical coordinates of each text region as a basis and the method of calling the trained word order correction model to sort the text regions.

[0066] S202, invoke the trained named entity recognition model to process the target text information and the contextual relationships of each character to obtain the entity label of each character.

[0067] In this embodiment, the trained named entity recognition model includes a trained encoder and a trained annotation module; therefore, the method of calling the trained named entity recognition model to process the target text information and the contextual relationships of each character to obtain the entity label of each character can be:

[0068] 1) Call the trained encoder:

[0069] ① Process the target text information and the contextual relationships of each character to determine the feature vector of each character;

[0070] ② Identify the first character preceding each character and the second character following each character in the target text information;

[0071] ③ Based on the feature vectors of each character, the feature vector of the first character, and the feature vector of the second character, each character is encoded to obtain the encoded characters.

[0072] Steps ①, ②, and ③ are all executed by the trained encoder.

[0073] Specifically, the trained encoder can be a pre-trained language model such as BERT (Bidirectional Encoder Representations from Transformer), XLNet (autoregressive and autoencoder model), or LayoutLM (Pre-training of Text and Layout for Document Image Understanding). Different pre-trained language models have different advantages. For example, BERT is simpler, while LayoutLM incorporates spatial information and performs better in more complex scenarios. Different pre-trained language models can be selected based on different needs, and there is no limitation here. The feature vector refers to the vector used to represent the features of each character. The feature vectors of the characters are extracted before the text information is input into the pre-trained language model.

[0074] In practice, when the trained encoder is the trained BERT model, the feature vectors of each character include word embeddings, segment embeddings, and position embeddings. Specifically, word embeddings can be determined by querying a word embedding table, segment embeddings are determined based on the global semantics of each character and the target text information, and position embeddings are determined by the contextual relationships of characters, since characters appearing in different positions in the text carry different semantics. The determination process of word embeddings, segment embeddings, and position embeddings is a technique commonly used by those skilled in the art and will not be elaborated here.

[0075] Since pre-trained language models employ a self-attention mechanism, after determining the feature vectors of each character, the trained BERT model needs to determine the first character preceding each character and the second character following each character in the target text information based on the contextual relationships between the characters. Then, by using the feature vectors of each character, the feature vector of the first character, and the feature vector of the second character, the semantic vector representation of each character is enhanced, thereby completing the encoding process of each character to obtain the encoded characters. The process of encoding each character by the trained BERT model is a technique commonly used by those skilled in the art and will not be elaborated here.

[0076] 2) Call the trained annotation module to annotate each encoded character and obtain the entity label for each character.

[0077] In this embodiment, the trained annotation module can be an annotation module obtained by training sequence annotation models such as Hidden Markov Model (HMM), Maximum Entropy Principle (MaxEnt), and BiLSTM-CRF (Bi-directional Long Short-Term Memory and Conditional Random Field). Furthermore, the entity label of each character can include label position and label type. The entity label of each character can vary depending on the preset type, and is not limited here. The preset type refers to the type of product; for example, if the product is braised eggplant, then the preset type is the dish name; if the product is Pilates, then the preset type is the fitness program.

[0078] Optionally, to further improve the efficiency of the product information entry system, entity tags can be set based on the product information required by the system. The information extracted from these entity tags can then be directly used by the system. For example, when the product is a dish, the dish name, price, currency unit, and unit of measurement are generally required. Therefore, entity tags can be set including B-term, I-term, B-price, I-price, B-currency, I-currency, B-unit, I-unit, and O. In this context, "term" refers to the dish name, such as shredded pork with garlic sauce, rock sugar pear, or honey grapefruit tea; "price" refers to the dish price; "currency" refers to the currency unit, such as yuan, US dollar, or Japanese yen; "unit" refers to the unit of measurement, such as jin, plate, serving, 300g, or position; "B," "I," and "O" indicate the label position, with "B" indicating the beginning and "I" indicating the interior. For example, a character with the entity label "B-term" means that the character is the first character of a dish name, and a character with the entity label "I-term" means that the character is an interior character of a dish name (i.e., a character other than the first character); and "O" refers to non-entity information, meaning that the character does not belong to any of the dish name, dish price, currency unit, or unit of measurement.

[0079] Optionally, the trained named entity recognition model can also be a pre-organized database storing field and field type correspondences. After obtaining the target text information, based on the target text information and the contextual relationship of each character, at least one field is determined in the target text information. Then, the field type matching each field is searched in the pre-defined database, where the field type is the entity label of each character in each field. Finally, fields belonging to the same product are determined and merged. The determination of fields belonging to the same product can be based on manually set rules. For example, it can be set that: if two consecutive fields in the target text information are both dish names, then these two dish names belong to two different dishes, and therefore these two fields do not need to be merged. If two consecutive fields in the target text information are both dish prices, then these two fields are merged. If two consecutive fields in the target text information are dish name and dish price respectively, then these two fields belong to the same dish, and therefore these two fields can be merged.

[0080] S203, in the entity tags of each character, determine at least one entity tag group, wherein the entity tags contained in each entity tag group have the same tag type.

[0081] In S204, the characters corresponding to each entity tag in the same entity tag group are combined to obtain product information.

[0082] In the embodiments of the present application, the manner of determining at least one entity tag group from the entity tags of each character may be: 1) If there is one target entity tag in the entity tags of each character, construct an entity tag group from the target entity tag and the entity tags having the same tag type as the target entity tag; 2) If there are multiple target entity tags in the entity tags of each character, determine a plurality of tag clusters, each tag cluster comprising the entity tags between any one target entity tag and the next target entity tag among the entity tags of each character; and construct an entity tag group from the entity tags having the same tag type as any target entity tag and the any target entity tag in each tag cluster.

[0083] Herein, the manner of determining the target entity tag from the entity tags of each character may be determined according to the tag position of the entity tag of each character. If the tag position of any entity tag is a preset tag position, the any entity tag is determined as the target entity tag.

[0084] In the embodiments of the present application, the product information refers to information related to a product. For example, if a target image shows a menu, the products in the menu are dishes, and the product information may include dish names, dish prices, currencies, specifications, and the like.

[0085] For example, please refer to the accom Figure 6 panying drawings Figure 6 which shows a schematic diagram of determining an entity tag group, where the tag positions can be preset as B and O. After acquiring the text information 601 "Cold Donkey Meat_60 Yuan / Jin", the text information 601 and the context relationship of each character in the target text information are input into a trained named entity recognition model (composed of a BERT model and a BiLSTM-CRF model), and then the trained named entity recognition model outputs the entity tag of each character. Wherein, the entity tag of the character "Liang" is "B-term", the entity tag of the character "Ban" is "I-term", the entity tag of the character "Lv" is "I-term", the entity tag of the character "Rou" is "I-term", the entity tag of the character "_" is "O", the entity tag of the character "6" is "B-price", the entity tag of the character "0" is "I-price", the entity tag of the character "Yuan" is "B-currency", the entity tag of the character " / " is "O", and the entity tag of the character "Jin" is "B-unit".

[0086] After obtaining the entity tags for each character, we can identify multiple target entity tags: "B-term", "O", "B-price", "B-currency", "O", and "B-unit". Then, we determine the tag clusters. Since the next target entity tag for "B-term" is "O", "B-term", "I-term", "I-term", and "I-term" form a tag cluster. Similarly, we can obtain other tag clusters, which will not be elaborated here. Next, since all entity tags in the tag cluster formed by "B-term", "I-term", "I-term", and "I-term" have the tag type "term", we can determine that "B-term", "I-term", "I-term", and "I-term" constitute entity tag group 1. Similarly, we can obtain entity tag groups 2 through 6. Finally, by combining the characters corresponding to the entity tags within the same entity tag group, we can obtain the product information. For example, the product information obtained by combining the characters corresponding to the entity tags in entity tag group 1 is "cold donkey meat". The final product information includes "cold donkey meat", "_", "60", "yuan", " / ", and "jin".

[0087] Optionally, since O refers to non-entity information, that is, unimportant information that is not very meaningful to product information, the tag position can also be pre-set as B. Then, multiple target entity tags can be identified as "B-term", "B-price", "B-currency", and "B-unit". The next target entity tag for "B-term" is then determined to be "B-price". Therefore, "B-term", "I-term", "I-term", "I-term", and "O" form a tag cluster. Since the tag type of "O" is different from that of "B-term", "I-term", "I-term", and "I-term", "B-term", "I-term", and "I-term" can be identified as an entity tag group. Similarly, "B-price" and "I-price" can be identified as an entity tag group, "B-currency" as an entity tag group, and "B-unit" as an entity tag group. Therefore, the final product information can include "cold donkey meat", "60", "yuan", and "jin".

[0088] Optionally, when outputting the product information, the product information may also be entered into the system correspondingly according to the label type of the entity label of the characters constituting the product information. Specifically, if the label type of the entity label of the characters "cold donkey meat" is dish name, then the cold donkey meat can be correspondingly entered into the dish name part of the system, obtaining "dish name: cold donkey meat". Similarly, the finally obtained product information entered into the system may be "dish name: cold donkey meat; dish price: 60; currency unit: yuan; measurement unit: jin".

[0089] In the embodiments of the present application, the target text information in the target image and the context relationship of each character in the target text information are acquired first; then the trained named entity recognition model is invoked to process the target text information and the context relationship of each character, so as to obtain the entity label of each character; next, at least one entity label group is determined from the entity labels of the characters, where the label types of the entity labels included in each entity label group are the same; finally, the characters corresponding to the entity labels in the same entity label group are combined to obtain the product information. In the embodiments of the present application, by invoking the trained named entity recognition model, each character in the target text information is labeled, after the entity label of each character is determined, the characters corresponding to the entity labels belonging to the same entity label group are combined to obtain the product information; since the target text information is processed in units of characters and then the corresponding characters are combined to obtain the product information, the problems such as inaccurate recognition caused by factors such as text content adhesion and diverse styles and designs of product information in the target image when the product information in the target image is directly recognized through text recognition are avoided, so that the product information in the image can be accurately extracted, which is further beneficial to improving the efficiency of entering the product information into the system.

[0090] Based on the above product information extraction solution, the embodiments of the present application provide another product information extraction method. Refer to Figure 7 , which is a schematic flowchart of another product information extraction method provided by the embodiments of the present application. Figure 7 The product information extraction method shown in the figure may be executed by a server or a terminal device. Figure 7 The product information extraction method shown in the figure may include the following steps:

[0091] S701: Acquire target text information in a target image, and the context relationship of each character in the target text information.

[0092] For the specific implementation of step S701, reference may be made to the specific implementation of step S201, which will not be repeated herein.

[0093] S702 calls the trained named entity recognition model to process the target text information and the contextual relationships of each character, and obtains the entity label of each character.

[0094] In this embodiment, the process of training the named entity recognition model can be as follows: 1) Obtaining a first training sample, which includes first training text information and target entity labels for each training character in the first training text information; 2) Performing semantic analysis on the first training text information to obtain the contextual relationship of each training character in the first training text information; 3) Calling the named entity recognition model to process the first training text information and the contextual relationship of each training character in the first training text information to obtain the predicted entity labels for each training character in the first training text information; 4) Training the named entity recognition model based on the predicted entity labels for each training character in the first training text information and the target entity labels for each training character in the first training text information to obtain the trained named entity recognition model.

[0095] Specifically, the first training text information refers to product-related text information, and the target entity labels of each training character in the first training text information can be manually annotated. The process of training the named entity recognition model based on the predicted entity labels of each training character in the first training text information and the target entity labels of each training character in the first training text information is a supervised learning training process. The specific training process is a technique commonly used by those skilled in the art and will not be elaborated here.

[0096] In one embodiment, the named entity recognition model includes an encoder and an annotation module; therefore, before training the named entity recognition model, it may also include:

[0097] 1) Obtain the second training sample, which includes the second training text information, which includes multiple training characters, and the information type of the second training text information is a preset type;

[0098] Specifically, the second training text information refers to product-related text information. The preset type can be the type of product indicated by the product-related text information contained in the second training text information. For example, if the product is braised eggplant or cold donkey meat, then the preset type can be the dish name; if the product is Pilates or swimming, then the preset type can be a fitness program.

[0099] 2) Perform semantic analysis on the second training text information to obtain the contextual relationships of each training character in the second training text information;

[0100] Specifically, performing semantic analysis on the second training text information to obtain the contextual relationships of each training character in the second training text information is a common technique, which will not be elaborated here.

[0101] 3) Generate target attention parameters based on each training character in the second training text information and the contextual relationship of each training character in the second training text information;

[0102] 4) Update the attention parameters in the encoder to the target attention parameters to obtain the updated encoder;

[0103] Specifically, the encoder employs a self-attention mechanism. If two encoders have different attention parameters in this mechanism, they will pay different levels of attention to the same characters within the same text, assigning different encoding weights and logic. Consequently, the encoded characters obtained by the two encoders will also differ. Therefore, by updating the attention parameters in the encoder to the target attention parameters, an updated encoder is obtained. This improves the encoder's representational ability in specific scenarios, resulting in more accurate character encoding and enhancing its robustness to entity boundary issues in Chinese contexts.

[0104] In specific implementation, in a menu scenario, training text information N, whose information type is dish type, can be obtained. This training text information N contains a large amount of information about the dish, such as its name, price, and specifications. Then, the BERT model is trained using this training text information N for vocabulary augmentation. Please refer to the appendix for details. Figure 8 The diagram illustrates an encoding process. After determining the feature vectors of each character / word in the training text information N, the feature vectors of each character / word are input into Self-Attention. Based on the target attention parameters, the processed feature vectors of each character / word are subjected to ADD (weighting) and Norm (normalization). The weighted and normalized data are then input into FNN (fully connected layer) for processing, and then subjected to another round of weighting and normalization to obtain the encoded characters / words. The encoded characters / words are then input into Linear (linear layer) or CRF (Conditional Random Field) for linear processing.

[0105] The calculation process for the target attention parameters in Self-Attention is as follows:

[0106] ① Create Spanx i and Spanx j The relative distance matrix.

[0107] Specifically, "span" refers to a segment of text content within an input document in the Named Entity Recognition (NER) task. For examples, please see the appendix. Figure 9 The diagram illustrates a Span, where the input text is "Nanjing Yangtze River Bridge". Based on this input text, Spans such as "Nanjing", "Nanjing City", "Mayor", "Yangtze River", "Yangtze River Bridge", and "Bridge" can be defined.

[0108] Where, x i Or x j It can be used to indicate individual characters or words in text information, such as Figure 8 As shown, in the menu scenario, x i Or x j This refers to terms like "Yunnan," "South," "Yunnan," and "Crossing-the-Bridge Rice Noodles." Additionally, such as... Figure 8 As shown, each Span is associated with a token, a head position, and a tail position. The token represents a character or a word, and the head and tail positions represent the positions of the first and last characters of the token within the string, respectively. Then, since the positional relationships between two different Spans include intersection, inclusion, and separation, Spanx can be constructed... i and Spanx j The relative distance matrix represents these relative positional relationships. The formula for the dense matrix relative positional relationship matrix is ​​as follows:

[0109]

[0110]

[0111]

[0112]

[0113] in, x represents i The head position and x j The distance between the head positions, x represents i The head position and x j The distance between the tail positions is calculated using the same formula as the other two formulas, which will not be elaborated here. This yields the relative distance matrix shown in Figure 801.

[0114] ② Calculate the dense matrix: Apply the ReLU activation function to the obtained relative distance matrix to obtain the dense matrix R. ijThe formula is as follows:

[0115]

[0116] Among them, W r This refers to a learnable parameter matrix that can... and Substituting into the following formula, we can calculate the result. and

[0117]

[0118]

[0119] Where k is the dimension index of the position encoding, 2k represents an even position, and 2k+1 refers to an odd position; the BERT model consists of multiple Transformers, d model The dimension output by Transformers can be 512, etc.; generally, d model =d head ×H num d head This refers to the dimension of each head, H. num The number of heads is a concept introduced by the multi-head mechanism of Transformers.

[0120] ③ Calculate the target attention weight, which is the matching degree between the query vector of a certain character / word and the key vectors corresponding to each character / word. The query vector and key vectors can be obtained from the feature vector of the current character / word. The attention matrix of the target attention weight can be obtained from the feature vector of the current character / word and the density value R. ij The formula is as follows:

[0121]

[0122] in, All of these are learnable parameters. Representation matrix W q The transpose of the matrix, Representation matrix The transpose of u T Let v be the transpose of u. T Denotes the transpose of v. This is the Attention matrix calculated for the current scene. x represents i The corresponding feature vector (Embedding vector), x representsj The corresponding feature vector (Embedding vector), R ij This is the dense matrix calculated in step ②.

[0123] ④ Update the attention parameters in the BERT model to the target attention parameters to obtain the updated BERT model. The update process formula is as follows, first using three matrices W... q W k W v The mapping yields Q, K, and V, and then the original formula... The calculated A i,j Updated to the result obtained in step ③ Finally, the attention parameters in the BERT model are updated to the target attention parameters using a normalization function:

[0124] [Q,K,V]=E x [W q W k W v ];

[0125]

[0126] Attention(A,V)=softmax(A)V;

[0127] Where Q refers to the query vector, K refers to the key vector, V refers to the value vector, and softmax is the normalization function. Both are learnable parameter matrices, where E represents a lookup table or the previous Transformer layer, and A... i,j This refers to the value of A at the coordinate (i,j), where A is A. i,j The matrix formed.

[0128] Since the BERT model can be composed of multiple Transformers, each Transformerlayer refers to 8 self-attention layers contained through the multi-head attention mechanism of the Transformer. Each self-attention layer has an output dimension of 64, so the feature dimension of each Transformer layer output is 512.

[0129] Optionally, the process of calling the named entity recognition model to process the first training text information and the contextual relationships of each training character in the first training text information to obtain the predicted entity labels of each training character in the first training text information can be as follows: 1) Call the updated BERT model to process the first training text information and the contextual relationships of each training character in the first training text information to determine the feature vector of each training character; and determine the third character before each training character and the fourth character after each training character in the first training text information; and based on the feature vectors of each training character in the first training text information, the feature vector of the third character of each training character in the first training text information, and the feature vector of the fourth character of each training character in the first training text information, encode each training character in the first training text information to obtain each encoded training character; 2) Call the annotation module to annotate each encoded training character to obtain the entity labels of each training character in the first training text information.

[0130] S703, in the entity tags of each character, at least one entity tag group is determined, and the entity tags contained in each entity tag group have the same tag type.

[0131] S704 combines the characters corresponding to the various entity tags in the same entity tag group to obtain product information.

[0132] The specific implementation methods of steps S703 to S704 can be found in the specific implementation methods of steps S203 to S204, and will not be repeated here.

[0133] S705, if the product information obtained based on the target text information contains target product information, and the target product information contains a target character that is different from all characters in the preset character set, then search for a character in the preset character set that semantically matches the target character.

[0134] S706, update the target character in the target product information with the found character to obtain the updated target product information.

[0135] In the embodiments of the present application, the character format of each character in the preset character set is uniform. For example, it can be set that the preset character set A contains all Chinese characters, then the characters such as the currency "¥", "RMB", "$" identified in the menu can be used as target characters that are all different from each character in the preset character set A; then, the character semantically matching the character "¥" in the preset character set A is "Yuan", the character semantically matching the character "RMB" is "Yuan", and the character semantically matching the character "$" is "US Dollar"; finally, the characters "¥", "RMB" and "$" in the product information will be updated to "Yuan", "Yuan" and "US Dollar" respectively.

[0136] In one embodiment, the manner of searching for a character semantically matching the target character in the preset character set may be manually sorting out the preset character set in advance, and marking similar words and keywords of each character in the preset character set to form a database, so that the character semantically matching the target character can be directly searched in the database. It is also possible to pre-train a synonym matching model, and search for the character semantically matching the target character in the preset character set through the trained synonym matching model; wherein the synonym matching model is a machine learning model, and the training process of the synonym matching model is a technical means commonly used by those skilled in the art, which will not be repeated here.

[0137] In one embodiment, it may also be: if there is target product information in the product information obtained based on the target text information, and the target product information does not meet the preset normalization rule, the target product information may be normalized to update the target product information, so as to obtain the updated target product information.

[0138] Specifically, the preset normalization rule refers to a rule that can unify the format of characters in the product information, which can be set manually or set by the system, and is not limited herein. For example, in the menu scenario, the preset normalization rules can be set as "removing trailing zeros of prices", "unifying traditional Chinese characters into simplified Chinese characters", "converting Chinese uppercase price numbers into Arabic numerals", and "unifying the currency representation into the format of x Yuan per plate", etc., so that the format of characters in the product information remains uniform, which is beneficial to improving the efficiency of the product information entry system.

[0139] For example, please refer to the attached Figure 10This diagram illustrates a process for extracting product information, where a preset normalization rule is set as "converting traditional Chinese characters to simplified Chinese characters." Menu image 1001 is uploaded to product information extraction system 1002. Product information extraction system 1002 first performs OCR recognition and word order correction on menu image 1001 to obtain target text information W in menu image 1001, as well as the contextual relationships of each character in target text information W. Then, product information extraction system 1002 uses a named entity recognition model to process the target text information W and the contextual relationships of each character, obtaining entity labels for each character. It then combines the characters corresponding to the entity labels in the same entity label group to obtain product information. At this point, all Chinese characters in the product information are in traditional Chinese. Finally, product information extraction system 1002 performs normalization processing on the product information, that is, updating the traditional Chinese characters in the product information to simplified Chinese characters according to the preset normalization rule, ultimately obtaining the updated product information 1003.

[0140] In this embodiment, the target text information in the target image and the contextual relationships of each character in the target text information are first obtained. Then, the trained named entity recognition model is called to process the target text information and the contextual relationships of each character to obtain the entity labels of each character. Next, at least one entity label group is determined from the entity labels of each character, and the characters corresponding to each entity label in the same entity label group are combined to obtain the product information. Finally, if the product information contains target product information, and the target product information contains a target character that is different from all characters in the preset character set, then a character that semantically matches the target character is searched in the preset character set; and the target character in the target product information is updated with the found character to obtain the updated target product information. This application embodiment uses a trained named entity recognition model to label each character in the target text information, determining the entity label for each character. Then, it combines the characters corresponding to entity labels belonging to the same entity label group to obtain product information. Because the target text information is processed character by character, and the corresponding characters are combined to obtain product information, it avoids the problems of inaccurate recognition caused by text content adhesion and the diverse styles and designs of product information in the target image when directly recognizing product information through text recognition. This allows for accurate extraction of product information from the image, thereby improving the efficiency of the product information entry system. Furthermore, by searching a preset character set for characters semantically matching the target character and updating the target character in the target product information with the found character, the format of the product information can be standardized, further improving the efficiency of the product information entry system.

[0141] Based on the above product information extraction scheme and system, this application provides a product information extraction method. See also... Figure 11 This is a flowchart illustrating a product information extraction method provided in an embodiment of this application. Figure 11 The product information extraction method shown can be obtained by Figure 1 The server and terminal devices shown are used for execution. Figure 11 The product information extraction method shown may include the following steps:

[0142] S1101, the terminal device sends the target image to the server.

[0143] In this embodiment of the application, the terminal device may send the target image to the server via wireless or wired communication, or via encrypted transmission, or via other means, which are not limited here.

[0144] S1102, the server obtains the target text information in the target image, as well as the contextual relationship of each character in the target text information.

[0145] S1103, the server calls the trained named entity recognition model to process the target text information and the contextual relationships of each character to obtain the entity label of each character.

[0146] S1104, the server determines at least one entity tag group in the entity tags of each character, and the entity tags contained in each entity tag group have the same tag type.

[0147] S1105, the server combines the characters corresponding to each entity tag in the same entity tag group to obtain product information.

[0148] The specific implementation methods of steps S1102 to S1105 can be found in the specific implementation methods of steps S201 to S204, and will not be repeated here.

[0149] S1106, the server sends product information to the terminal device.

[0150] The method of sending product information to the terminal device in step S1106 is a technical means commonly used by those skilled in the art, and will not be described in detail here.

[0151] In this embodiment, the target text information and the contextual relationships of each character in the target text information are first obtained from the target image. Then, the trained named entity recognition model is called to process the target text information and the contextual relationships of each character to obtain the entity labels of each character. Next, at least one entity label group is determined from the entity labels of each character, wherein the entity labels contained in each entity label group have the same label type. Finally, the characters corresponding to each entity label in the same entity label group are combined to obtain product information. This embodiment of the application calls the trained named entity recognition model to annotate each character in the target text information, determines the entity labels of each character, and then combines the characters corresponding to each entity label belonging to the same entity label group to obtain product information. Since the target text information is processed character by character, and the corresponding characters are combined to obtain product information, the problems of inaccurate recognition caused by text content adhesion and the variety of styles and designs of product information in the target image that occur when directly recognizing product information in the target image through text recognition are avoided. This enables accurate extraction of product information from the image, thereby improving the efficiency of the product information entry system.

[0152] Based on the above embodiments of the product information extraction method, this application provides a product information extraction device. See also... Figure 12 This is a schematic diagram of a product information extraction device provided in an embodiment of this application. The product information extraction device may include an acquisition unit 1201 and a processing unit 1202. Figure 12 The product information extraction device shown can operate the following units:

[0153] The acquisition unit 1201 is used to acquire target text information in the target image, as well as the contextual relationship of each character in the target text information;

[0154] The processing unit 1202 is also used to call the trained named entity recognition model to process the target text information and the contextual relationship of each character to obtain the entity label of each character.

[0155] The processing unit 1202 is further configured to determine at least one entity tag group in the entity tags of each character, wherein the entity tags contained in each entity tag group have the same tag type.

[0156] The processing unit 1202 is also used to combine the characters corresponding to each entity label in the same entity label group to obtain product information.

[0157] In one embodiment, the trained named entity recognition model includes a trained encoder and a trained annotation module;

[0158] The processing unit 1202 is further configured to: call the trained encoder to process the target text information and the contextual relationship of each character, and determine the feature vector of each character; determine the first character before each character and the second character after each character in the target text information; encode each character based on the feature vector of each character, the feature vector of the first character, and the feature vector of the second character to obtain each encoded character; and call the trained annotation module to annotate each encoded character to obtain the entity label of each character.

[0159] In one embodiment, the acquisition unit 1201 is further configured to: perform text detection processing on the target image to obtain at least one text region and the position information of each text region in the target image; sort the at least one text region based on the position information of each text region in the target image to obtain sorted text regions; and concatenate the characters contained in each text region in the sorted text regions to obtain target text information.

[0160] In one embodiment, the location information of each text region in the target image includes the horizontal and vertical coordinates of each text region in the target image;

[0161] The acquisition unit 1201 is further configured to: determine at least one text region queue based on the ordinate of each text region, wherein the ordinate of the text region in the target text region queue of the at least one text region queue is less than the ordinate of the text region in the next text region queue of the target text region queue, the target text region queue is any text region queue in the at least one text region queue, and the ordinates of each text region in the same text region queue are matched; traverse at least one text region queue, and sort each text region in the currently traversed text region queue based on the abscissa of each text region in the currently traversed text region queue to obtain the sorted text regions.

[0162] In one embodiment, the acquisition unit 1201 is further configured to invoke a word order correction model, perform word order correction processing on at least one text region based on the position information of each text region, and obtain a processed text region sequence. The word order between characters in any text region of the processed text region sequence and characters in the previous text region of any text region, as well as characters in the next text region of any text region, satisfies a preset word order rule; the processed text region sequence is used as the sorted text region.

[0163] In one embodiment, the processing unit 1202 is further configured to: acquire a first training sample, the first training sample including first training text information and target entity labels for each training character in the first training text information; perform semantic analysis on the first training text information to obtain the contextual relationship of each training character in the first training text information; call a named entity recognition model to process the first training text information and the contextual relationship of each training character in the first training text information to obtain predicted entity labels for each training character in the first training text information; and train the named entity recognition model based on the predicted entity labels of each training character in the first training text information and the target entity labels of each training character in the first training text information to obtain the trained named entity recognition model.

[0164] In one embodiment, the named entity recognition model includes an encoder and an annotation module; the processing unit 1202 is further configured to: acquire a second training sample, the second training sample including second training text information, the second training text information including multiple training characters, and the information type of the second training text information being a preset type; perform semantic analysis on the second training text information to obtain the contextual relationship of each training character in the second training text information; generate target attention parameters based on each training character in the second training text information and the contextual relationship of each training character in the second training text information; update the attention parameters in the encoder to the target attention parameters to obtain an updated encoder;

[0165] The processing unit 1202 is further configured to: call the updated encoder to process the first training text information and the contextual relationship of each training character in the first training text information, and determine the feature vector of each training character; determine the third character before each training character and the fourth character after each training character in the first training text information; based on the feature vector of each training character in the first training text information, the feature vector of the third character of each training character in the first training text information, and the feature vector of the fourth character of each training character in the first training text information, encode each training character in the first training text information to obtain each encoded training character; and call the annotation module to annotate each encoded training character to obtain the entity label of each training character in the first training text information.

[0166] In one embodiment, the processing unit 1202 is further configured to: if the product information obtained based on the target text information contains target product information, and the target product information contains a target character that is different from all characters in the preset character set, then search for a character that semantically matches the target character in the preset character set; update the target character in the target product information with the found character, and obtain the updated target product information.

[0167] In one embodiment, the processing unit 1202 is further configured to: if there is a target entity tag in the entity tags of each character, construct an entity tag group consisting of entity tags of the same tag type as the target entity tag and the target entity tag; if there are multiple target entity tags in the entity tags of each character, determine multiple tag clusters, each tag cluster including entity tags between any target entity tag in the entity tags of each character and the next target entity tag of any target entity tag; and construct an entity tag group consisting of entity tags of the same tag type as any target entity tag in each tag cluster and any target entity tag.

[0168] According to one embodiment of this application, Figure 2 , Figure 7 as well as Figure 10 The steps involved in the product information extraction method shown can be derived from... Figure 12 The product information extraction device shown is executed by each unit within it.

[0169] According to another embodiment of this application, Figure 12 The product information extraction device shown can be composed of individual or combined units into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above units are based on logical function division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the product information extraction device based on logical function division may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0170] According to another embodiment of this application, the following can be achieved by running on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), a device capable of performing operations such as... Figure 2 , Figure 6 as well as Figure 9The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 12 The product information extraction apparatus shown herein, and the product information extraction method for implementing the embodiments of this application, are described. The computer program may be recorded on, for example, a computer-readable storage medium, loaded onto the aforementioned computing device via the computer-readable storage medium, and run therein.

[0171] In this embodiment, the target text information and the contextual relationships of each character in the target text information are first obtained from the target image. Then, the trained named entity recognition model is called to process the target text information and the contextual relationships of each character to obtain the entity labels of each character. Next, at least one entity label group is determined from the entity labels of each character, wherein the entity labels contained in each entity label group have the same label type. Finally, the characters corresponding to each entity label in the same entity label group are combined to obtain product information. This embodiment of the application calls the trained named entity recognition model to annotate each character in the target text information, determines the entity labels of each character, and then combines the characters corresponding to each entity label belonging to the same entity label group to obtain product information. Since the target text information is processed character by character, and the corresponding characters are combined to obtain product information, the problems of inaccurate recognition caused by text content adhesion and the variety of styles and designs of product information in the target image that occur when directly recognizing product information in the target image through text recognition are avoided. This enables accurate extraction of product information from the image, thereby improving the efficiency of the product information entry system.

[0172] Based on the above method and apparatus embodiments, this application also provides an electronic device. See also Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 13 The electronic device shown may include at least a processor 1301, an input interface 1302, an output interface 1303, and a computer storage medium 1304. The processor 1301, input interface 1302, output interface 1303, and computer storage medium 1304 may be connected via a bus or other means.

[0173] Computer storage medium 1304 can be stored in the memory of electronic device. Computer storage medium 1304 is used to store computer programs, which include program instructions. Processor 1301 is used to execute the program instructions stored in computer storage medium 1304. Processor 1301 (or CPU (Central Processing Unit)) is the computing and control core of electronic device. It is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to realize the above-mentioned product information extraction method flow or corresponding functions.

[0174] This application embodiment also provides a computer storage medium (Memory), which is a memory device in an electronic device used to store programs and data. It is understood that the computer storage medium here can include the built-in storage medium in a terminal, or it can include an extended storage medium supported by the terminal. The computer storage medium provides storage space, which stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by the processor 1301. These instructions can be one or more computer programs (including program code). It should be noted that the computer storage medium here can be a high-speed random access memory (RAM), or it can be non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer storage medium located remotely from the aforementioned processor.

[0175] In one embodiment, the processor 1301 may load and execute one or more instructions stored in the computer storage medium to implement the aforementioned related... Figure 2 , Figure 7 as well as Figure 10 In the product information extraction method embodiment, the corresponding steps of the method are as follows: In a specific implementation, one or more instructions in the computer storage medium are loaded and executed by the processor 1301:

[0176] The processor 1301 acquires the target text information in the target image, as well as the contextual relationship of each character in the target text information;

[0177] The processor 1301 calls the trained named entity recognition model to process the target text information and the contextual relationship of each character to obtain the entity label of each character;

[0178] The processor 1301 determines at least one entity tag group in the entity tags of each character, wherein the entity tags contained in each entity tag group have the same tag type.

[0179] The processor 1301 combines the characters corresponding to each entity label in the same entity label group to obtain product information.

[0180] In one embodiment, the trained named entity recognition model includes a trained encoder and a trained annotation module; the processor 1301 calls the trained encoder to process the target text information and the context relationship of each character to determine the feature vector of each character; determines the first character before each character and the second character after each character in the target text information; and encodes each character based on the feature vector of each character, the feature vector of the first character, and the feature vector of the second character to obtain each encoded character.

[0181] The processor 1301 calls the trained annotation module to annotate each encoded character and obtain the entity label for each character.

[0182] In one embodiment, the processor 1301 performs text detection processing on the target image to obtain at least one text region and the position information of each text region in the target image;

[0183] The processor 1301 sorts at least one text region based on the position information of each text region in the target image to obtain sorted text regions;

[0184] The processor 1301 concatenates the characters contained in each text region of the sorted text region to obtain the target text information.

[0185] In one embodiment, the location information of each text region in the target image includes the horizontal and vertical coordinates of each text region in the target image;

[0186] Processor 1301 sorts at least one text region based on the position information of each text region in the target image, obtaining sorted text regions, including:

[0187] The processor 1301 determines at least one text region queue based on the ordinate of each text region. The ordinate of the text region in the target text region queue of the at least one text region queue is less than the ordinate of the text region in the next text region queue of the target text region queue. The target text region queue is any text region queue in the at least one text region queue. The ordinates of each text region in the same text region queue are matched.

[0188] The processor 1301 traverses at least one text region queue, sorts each text region in the currently traversed text region queue based on the x-coordinate of each text region in the currently traversed text region queue, and obtains the sorted text regions.

[0189] In one embodiment, the processor 1301 sorts at least one text region based on the position information of each text region in the target image to obtain sorted text regions, including:

[0190] The processor 1301 calls the word order correction model and performs word order correction processing on at least one text region based on the position information of each text region to obtain a processed text region sequence. The word order between the characters in any text region of the processed text region sequence and the characters in the previous text region of any text region, as well as the characters in the next text region of any text region, satisfies the preset word order rules.

[0191] The processor 1301 uses the processed text region sequence as the sorted text region.

[0192] In one embodiment, the processor 1301 acquires a first training sample, which includes first training text information and target entity labels for each training character in the first training text information.

[0193] The processor 1301 performs semantic analysis on the first training text information to obtain the contextual relationship of each training character in the first training text information;

[0194] The processor 1301 calls the named entity recognition model to process the first training text information and the contextual relationship of each training character in the first training text information to obtain the predicted entity label of each training character in the first training text information.

[0195] The processor 1301 trains the named entity recognition model based on the predicted entity labels of each training character in the first training text information and the target entity labels of each training character in the first training text information, and obtains the trained named entity recognition model.

[0196] In one embodiment, the named entity recognition model includes an encoder and a labeling module;

[0197] Processor 1301 acquires a second training sample, the second training sample includes second training text information, the second training text information includes multiple training characters, and the information type of the second training text information is a preset type;

[0198] The processor 1301 performs semantic analysis on the second training text information to obtain the contextual relationship of each training character in the second training text information;

[0199] The processor 1301 generates target attention parameters based on each training character in the second training text information and the contextual relationship of each training character in the second training text information.

[0200] Processor 1301 updates the attention parameters in the encoder to the target attention parameters, thus obtaining the updated encoder;

[0201] Processor 1301 invokes the named entity recognition model to process the first training text information and the contextual relationships of each training character in the first training text information, to obtain the predicted entity labels of each training character in the first training text information, including:

[0202] The processor 1301 calls the updated encoder to process the first training text information and the contextual relationship of each training character in the first training text information to determine the feature vector of each training character.

[0203] The processor 1301 determines the third character that precedes each training character and the fourth character that follows each training character in the first training text information;

[0204] The processor 1301 encodes each training character in the first training text information based on the feature vectors of each training character in the first training text information, the feature vector of the third character of each training character in the first training text information, and the feature vector of the fourth character of each training character in the first training text information, to obtain each encoded training character.

[0205] The processor 1301 calls the annotation module to annotate each encoded training character to obtain the entity label of each training character in the first training text information.

[0206] In one embodiment, if the product information obtained based on the target text information contains target product information, and the target product information contains a target character that is different from all characters in the preset character set, then the processor 1301 searches for a character in the preset character set that semantically matches the target character.

[0207] The processor 1301 updates the target character in the target product information with the found character to obtain the updated target product information.

[0208] In one embodiment, the processor 1301 determines at least one entity tag group in the entity tags of each character, wherein the entity tag groups contain entity tags of the same tag type, including:

[0209] If there is a target entity tag in the entity tags of each character, the processor 1301 will construct an entity tag group with the entity tags of the same tag type as the target entity tag and the target entity tag;

[0210] If there are multiple target entity tags in the entity tags of each character, the processor 1301 determines multiple tag clusters, each tag cluster including the entity tags between any target entity tag in the entity tags of each character and the next target entity tag of any target entity tag;

[0211] The processor 1301 constructs an entity tag group from the entity tags in each tag cluster that have the same tag type as any target entity tag and any target entity tag.

[0212] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the aforementioned actions. Figure 2 , Figure 6 as well as Figure 9 The method embodiment shown. The computer-readable storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0213] It should be noted that the product information extraction scheme mentioned in this application is introduced using the relevant scenario of extracting dish information from the menu as an example, and it does not limit the scope of this application. The product information extraction mentioned in this application can also be applied to other scenarios that require extracting product information from images, and this application does not limit it.

[0214] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for extracting product information, characterized in that, include: Text detection processing is performed on the target image to obtain at least one text region and the position information of each text region in the target image; The word order rule is determined based on the word order of the characters in the product information corresponding to the target image; The word order correction model is invoked, and based on the positional information of each text region, the characters in each text region are concatenated to obtain concatenated text information. The word order correction model is then invoked to adjust the word order of the characters in the concatenated text information to obtain adjusted concatenated text information. At least one text region is sorted according to the characters in the adjusted concatenated text information to obtain a processed text region sequence. The word order between the characters in any text region of the processed text region sequence and the characters in the previous and next text regions of any text region satisfies the word order rules. If the target image is an image containing a menu, the word order adjustment includes adjusting the word order of the concatenated text information according to the category of the dish and the order of the dish names belonging to that category. The characters contained in each text region in the processed text region sequence are concatenated to obtain target text information; wherein, at least one text region includes multiple characters, and the target text information is the text information in the target image used to indicate product content; Obtain the contextual relationships of each character in the target text information; The trained named entity recognition model is invoked to process the target text information and the contextual relationship of each character to obtain the entity label of each character; If there are multiple target entity tags in the entity tags of each character, then multiple tag clusters are determined. Each tag cluster includes entity tags between any target entity tag in the entity tags of each character and the next target entity tag of that target entity tag. The multiple target entity tags include entity tags that indicate the tag positions of non-entity information. Entity tags in each tag cluster that have the same tag type as any target entity tag, along with any target entity tag, are combined into an entity tag group; The product information is obtained by combining the characters corresponding to the various entity tags in the same entity tag group; If the product information contains a target product information with a target character that is different from all characters in the preset character set, then search the preset character set for a character that semantically matches the target character. The target character is updated with the found character, and the tag type of the entity tag is added to the characters that make up the product information to obtain the updated target product information. The updated target product information meets the preset normalization rules, which include font conversion rules, and the layout of the updated target product information is different from the layout of the target image.

2. The method according to claim 1, characterized in that, The trained named entity recognition model includes a trained encoder and a trained annotation module; The trained named entity recognition model is invoked to process the target text information and the contextual relationships of each character to obtain entity labels for each character, including: The trained encoder is invoked to process the target text information and the contextual relationships of each character to determine the feature vector of each character. Determine the first character preceding each character and the second character following each character in the target text information; Based on the feature vectors of each character, the feature vector of the first character, and the feature vector of the second character, each character is encoded to obtain each encoded character; The trained annotation module is invoked to annotate each encoded character, thereby obtaining the entity label for each character.

3. The method according to claim 1, characterized in that, The method further includes: Obtain a first training sample, which includes first training text information and target entity labels for each training character in the first training text information. Semantic analysis is performed on the first training text information to obtain the contextual relationships of each training character in the first training text information; The named entity recognition model is invoked to process the first training text information and the contextual relationship of each training character in the first training text information to obtain the predicted entity label of each training character in the first training text information. Based on the predicted entity labels of each training character in the first training text information and the target entity labels of each training character in the first training text information, the named entity recognition model is trained to obtain the trained named entity recognition model.

4. The method according to claim 3, characterized in that, The named entity recognition model includes an encoder and a labeling module; The method further includes: Obtain a second training sample, the second training sample includes second training text information, the second training text information includes multiple training characters, and the information type of the second training text information is a preset type; Semantic analysis is performed on the second training text information to obtain the contextual relationships of each training character in the second training text information; Based on each training character in the second training text information and the contextual relationship of each training character in the second training text information, target attention parameters are generated; The attention parameters in the encoder are updated with the target attention parameters to obtain the updated encoder; The method of calling the named entity recognition model processes the first training text information and the contextual relationships of each training character in the first training text information to obtain the predicted entity label of each training character in the first training text information, including: The updated encoder is invoked to process the first training text information and the contextual relationships of each training character in the first training text information to determine the feature vector of each training character. Determine the third character preceding each training character and the fourth character following each training character in the first training text information; Based on the feature vectors of each training character in the first training text information, the feature vector of the third character of each training character in the first training text information, and the feature vector of the fourth character of each training character in the first training text information, each training character in the first training text information is encoded to obtain each encoded training character. The annotation module is invoked to annotate each encoded training character to obtain the entity label of each training character in the first training text information.

5. A product information extraction device, characterized in that, The product information extraction device includes an acquisition unit and a processing unit, wherein: The acquisition unit is used to perform text detection processing on the target image to obtain at least one text region and the position information of each text region in the target image; determine the word order rule based on the word order of characters in the product information corresponding to the target image; call the word order correction model to concatenate the characters in each text region based on the position information of each text region to obtain concatenated text information; call the word order correction model to adjust the word order of the characters in the concatenated text information to obtain adjusted concatenated text information; and sort at least one text region according to the characters in the adjusted concatenated text information to obtain a processed text region sequence. The word order of characters in any text region of the text region sequence satisfies the word order rule with respect to characters in the preceding and following text regions. The characters contained in each text region of the processed text region sequence are concatenated to obtain target text information. At least one text region includes multiple characters, and the target text information is text information in the target image used to indicate product content. If the target image is an image containing a menu, the word order adjustment includes adjusting the word order of the concatenated text information according to the category of the dish and the order of the dish names belonging to that category. The acquisition unit is also used to acquire the contextual relationship of each character in the target text information; The processing unit is also used to call the trained named entity recognition model to process the target text information and the contextual relationship of each character to obtain the entity label of each character; The processing unit is further configured to, if there are multiple target entity tags in the entity tags of each character, determine multiple tag clusters, each tag cluster including entity tags between any target entity tag in the entity tags of each character and the next target entity tag of any target entity tag; the multiple target entity tags include entity tags that indicate the tag position of non-entity information; and construct an entity tag group by combining entity tags of the same tag type as any target entity tag in each tag cluster with any target entity tag. The processing unit is also used to combine the characters corresponding to each entity label in the same entity label group to obtain product information; The processing unit is further configured to: if there is target product information in the product information that contains a target character that is different from all characters in the preset character set, then search for a character in the preset character set that semantically matches the target character; update the target character to the found character, and supplement the tag type of the entity tag for the characters that make up the product information, thereby obtaining the updated target product information; the updated target product information satisfies the preset normalization rules, the preset normalization rules include font conversion rules, and the layout of the updated target product information is different from the layout of the target image.

6. A computer device, characterized in that, It includes a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it implements the product information extraction method as described in any one of claims 1-4.

7. A computer storage medium, characterized in that, The computer storage medium stores computer program instructions, which, when executed by a processor, are used to perform the product information extraction method as described in any one of claims 1-4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1-4.

Images