Article identification method and device, storage medium and electronic equipment

Abstract

The invention provides an article identification method and device, a storage medium and electronic equipment. The method comprises the following steps: acquiring image data corresponding to an article package, identifying text data in the image data, performing feature coding on the text data to obtain language description text features corresponding to the text data, and storing the language description text features in the image data; the method comprises the steps of obtaining image data, obtaining visual features of the image data, generating image description text features of the image data based on the visual features, determining an article identifier corresponding to the article package based on the language description text features and the image description text features, and obtaining article identification data corresponding to the article identifier. According to the method, the convenience and accuracy of article identification are improved.

Description

Technical Field

[0001] The present specification relates to the field of computer technology, and more specifically, to an object identification method, device, storage medium and electronic device in the field of computer technology. Background Art

[0002] Nowadays, with the continuous development of information technology, users have more and more ways to obtain data through terminal devices. However, when identifying objects, in order to determine the relevant information of the objects, users are required to detect according to the characteristics of the objects, which results in low retrieval efficiency and insufficient accuracy. Summary of the invention

[0003] The present specification provides an object identification method, device, storage medium and electronic device. The method enables a user to photograph an object packaging through a terminal device, and obtain object identification data corresponding to the object packaging based on the language description text features and image description text features corresponding to the photographed image data, thereby achieving the effect of identifying the object packaging based on the combination of text and image, thereby improving the convenience and accuracy of object identification.

[0004] In a first aspect, a method for identifying an object is provided, the method comprising:

[0005] Acquire image data corresponding to the article packaging, and identify text data in the image data;

[0006] Performing feature encoding on the text data to obtain language description text features corresponding to the text data;

[0007] Acquire visual features of the image data, and generate image description text features of the image data based on the visual features;

[0008] An article identifier corresponding to the article packaging is determined based on the language description text feature and the image description text feature, and article identification data corresponding to the article identifier is acquired.

[0009] Through the above technical scheme, it is possible to obtain text data of the image data based on the image data of the identified article packaging, extract the language description text features corresponding to the text data, and the image description text features corresponding to the image data, and based on the language description text features and the image description text features, it is possible for a user to take a photo of the article packaging through a terminal device, and obtain the article identification data corresponding to the article packaging based on the language description text features and the image description text features corresponding to the photographed image data, thereby achieving the effect of identifying the article packaging based on the combination of text and image, thereby improving the convenience and accuracy of article identification.

[0010] In conjunction with the first aspect, in some possible implementations, acquiring visual features of the image data, and generating image description text features of the image data based on the visual features includes:

[0011] The image data is divided into at least one image block, and features are extracted from each image block to obtain a visual feature, and image-related features corresponding to each visual feature are determined based on the position of each image block in the image data;

[0012] Feature mapping is performed based on the image association features and each of the visual features to obtain image description text features corresponding to the image data.

[0013] Through the above technical solution, the visual features corresponding to the image data are extracted and converted into image description text features, thereby obtaining image description text features that accurately describe the image data, so that the object packaging in the image data can be identified according to the image description text features, thereby improving the accuracy of object identification.

[0014] In combination with the first aspect and the above implementation manner, in some possible implementation manners, dividing the image data into at least one image block, and extracting features from each image block to obtain visual features includes:

[0015] Segmenting the image data based on a preset segmentation size to obtain at least one image block;

[0016] A feature encoder is used to extract features from each of the image blocks to obtain visual features corresponding to each of the image blocks.

[0017] In combination with the first aspect and the above implementation manner, in some possible implementation manners, segmenting the image data based on a preset segmentation size to obtain at least one image block includes:

[0018] Acquiring an image size of the image data, scaling the image data based on a preset segmentation size and the image size, and obtaining a scaled image corresponding to the image data;

[0019] The scaled image is segmented based on the segmentation size to obtain at least one image block corresponding to the image data.

[0020] In combination with the first aspect and the above implementation manner, in some possible implementation manners, determining the item identifier corresponding to the item packaging based on the language description text feature and the image description text feature, and acquiring the item identification data corresponding to the item identifier includes:

[0021] Inputting the language description text features and the image description text features into a recognition model, and obtaining an item identification corresponding to the item packaging based on the recognition model, wherein the recognition model is pre-trained based on the item type of the item packaging and sample data corresponding to the item type;

[0022] The identification model is used to search in an article database based on the article identifier to obtain article identification data corresponding to the article packaging.

[0023] Through the above technical solution, the language description text features and image description text features corresponding to the image data are combined for recognition to obtain the item identification and item identification data corresponding to the item packaging, thereby improving the accuracy of item recognition based on dual recognition of language and image.

[0024] In combination with the first aspect and the above implementation manner, in some possible implementation manners, after acquiring the item identification data corresponding to the item identifier, the method further includes:

[0025] Generate the article introduction data corresponding to the article packaging by using the recognition model based on the article identification and the article identification data;

[0026] Output the item introduction data.

[0027] Through the above technical solution, item introduction data is generated according to the item identification and item recognition data layer, and the item introduction data is output, so that users can view the data introducing the item packaging in the image data, thereby improving the convenience of users in identifying items.

[0028] In combination with the first aspect and the above implementation manner, in some possible implementation manners, if a query statement for the article packaging is obtained, feature encoding is performed on the query statement to obtain a query feature corresponding to the query statement;

[0029] Reply data is generated and output based on the query characteristics and the item identification data.

[0030] Through the above technical solution, response data corresponding to the query statement is generated according to the query statement input by the user and the recognition result of the item packaging in the image data, so as to answer the user's query statement, facilitate the user to identify and understand the item corresponding to the item packaging, and improve the accuracy and convenience of item recognition.

[0031] In a second aspect, an object identification device is provided, the device comprising:

[0032] A data acquisition unit, used to acquire image data corresponding to the article packaging, and identify text data in the image data;

[0033] A feature acquisition unit, used for performing feature encoding on the text data to obtain language description text features corresponding to the text data;

[0034] The feature acquisition unit is further used to acquire visual features of the image data, and generate image description text features of the image data based on the visual features;

[0035] The article identification unit is used to determine the article identification corresponding to the article packaging based on the language description text feature and the image description text feature, and obtain the article identification data corresponding to the article identification.

[0036] According to a third aspect, a computer program product is provided, comprising: a computer program code, which, when executed on a computer, enables the computer to execute the method according to the first aspect or any possible implementation of the first aspect.

[0037] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program code. When the computer program code runs on a computer, the computer executes the method in the above-mentioned first aspect or any possible implementation manner of the first aspect.

[0038] According to a fifth aspect, an electronic device is provided, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a system architecture diagram of an object identification method provided by an embodiment of this specification;

[0040] Figure 2 It is a flowchart of an object identification method provided in an embodiment of this specification;

[0041] Figure 3 It is a flowchart of an object identification method provided in an embodiment of this specification;

[0042] Figure 4 is an example schematic diagram of image data provided by an embodiment of this specification;

[0043] Figure 5 is an example schematic diagram of image segmentation provided in an embodiment of this specification;

[0044] Figure 6 is an example schematic diagram of an image description text feature provided by an embodiment of this specification;

[0045] Figure 7This is an example diagram of an item introduction data provided in the embodiment of this specification

[0046] Figure 8 It is a structural schematic diagram of an object identification device provided in an embodiment of this specification;

[0047] Fig. 9 It is a structural schematic diagram of an object identification device provided in an embodiment of this specification;

[0048] Fig.10 It is a structural schematic diagram of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0049] The technical solutions in this specification will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of this specification, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of this specification, "multiple" means two or more than two.

[0050] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0051] Figure 1 is a system architecture diagram of an object identification method provided in an embodiment of this specification. Figure 1 As shown, the object identification method provided in the embodiment of this specification can be applied to a terminal device to realize the process of identifying an object according to an object packaging. The system structure provided in the embodiment of this specification mainly includes a terminal device 10, an object packaging 20, a recognition model 30, and an object identification data 40. Among them, the terminal device 10 can be a device with an image shooting function, such as a personal computer, a smart phone, a tablet computer, etc.; the object packaging 20 can be a packaging for packaging an object, such as a packaging box or a packaging bag; the recognition model 30 can be a large language model with a text feature recognition function, which can be set in the terminal device 10 or in a server corresponding to the terminal device 10, and a model for object identification through data transmission between the terminal device 10 and the server, which can be set according to actual conditions; the object identification data 40 can be data generated by the recognition model 30 for identifying the object packaging.

[0052] In the related art, when identifying an object, the method used is to have the user search for the feature words of the object, manually screen the retrieved data, and obtain the identification result of the object, resulting in insufficient recognition efficiency and low accuracy of object recognition.

[0053] According to the embodiments of the present specification, a user photographs an article packaging 20 through a terminal device 10, obtains image data corresponding to the article packaging 20, identifies text data in the image data, performs feature encoding on the text data to obtain language description text features corresponding to the text data, obtains visual features of the image data, generates image description text features of the image data based on the visual features, uses a recognition model 30 to determine an article identification corresponding to the article packaging based on the language description text features and the image description text features, and obtains article identification data 40 corresponding to the article identification, thereby enabling a user to photograph an article packaging through a terminal device, obtain article identification data corresponding to the article packaging based on the language description text features and the image description text features corresponding to the photographed image data, achieves recognition based on a combination of text and image, and realizes the effect of identifying the article packaging, thereby improving the convenience and accuracy of article identification.

[0054] based on Figure 1 The system architecture shown below will be combined with Figure 2-Figure 7 , the object identification method provided in the embodiments of this specification is introduced in detail.

[0055] See also Figure 2 , which is a flow chart of an object identification method provided in the embodiment of this specification. Figure 2 As shown, the method of the embodiment of this specification may include the following steps S102 to S108.

[0056] S102, acquiring image data corresponding to the article packaging, and identifying text data in the image data;

[0057] In one embodiment, image data corresponding to article packaging is obtained. The article packaging may be packaging used to package and protect articles to facilitate transportation and sale of the articles, such as a packaging box or a packaging bag. The image data may be data obtained by photographing the article packaging. The article packaging in the image data is identified to obtain text data in the article packaging. The text data may be data obtained by identifying the image data according to a text recognition model.

[0058] Specifically, the method for acquiring image data may be: data obtained by a user photographing the article packaging through a camera component of a terminal device, and the article packaging of the image data may be any side of the article packaging. It is understandable that in order to improve the accuracy of identifying article packaging, a feasible method is to obtain data by photographing the front of the article packaging, wherein the front of the article packaging may be one of the sides of the article packaging pre-set during the design and production process of the article packaging. Exemplarily, by running the "Scan" function of the terminal device, the camera component of the terminal device is controlled to photograph the article packaging, thereby obtaining image data corresponding to the article packaging. It is understandable that the image data may also be obtained by running other functions of the terminal device, and the method used to obtain the image data may be specifically set according to the actual situation.

[0059] It should be noted that if the image data includes more than one object, each object in each image data can be selected based on the attention mechanism to determine the packaging of the object in the image data.

[0060] The text data may be text data such as the name of the item included in the item packaging, or text data corresponding to the icon on the item packaging, obtained by extracting text from the image data using a text extraction model. The text extraction model may be a model with a text extraction function, such as OCR (Optical Character Recognition).

[0061] S104, feature encoding the text data to obtain language description text features corresponding to the text data;

[0062] In one embodiment, after acquiring the text data in the image data, the text data is encoded based on a text encoder to obtain language description text features that can be recognized by the model. The language description text features can be feature data that can be recognized by the model after the text data is formatted, so that the language description text features can be subsequently recognized to obtain the language description corresponding to the item packaging.

[0063] Among them, the encoder used to convert text data into language description text features can be a Tokenizer encoder, which has the function of converting text data of natural language text into language description text features in digital vector format.

[0064] S106, acquiring visual features of the image data, and generating image description text features of the image data based on the visual features;

[0065] In one embodiment, after feature extraction and data format conversion of image data, visual features corresponding to the image data are obtained. The visual features may be features used to characterize the visual perception of the user, such as gradient features, color features, and texture features. Recognition is performed based on the visual features to generate image description text features corresponding to the image data. The image description text features may be features used to describe the packaging of items in the image data.

[0066] Among them, the gradient feature may be edge and contour information representing image data, the color feature may be a color histogram, a color set, etc. representing an image, and the texture feature may be a surface property representing an image.

[0067] It should be noted that the execution order of obtaining the language description text features according to step S104 and obtaining the image description text features according to step S106 can be set according to actual needs. Step S104 can be executed first to obtain the language description text features, and then step S106 can be executed to obtain the image description text features; step S106 can also be executed first to obtain the image description text features, and then step S104 can be executed to obtain the language description text features; step S104 and step S106 can also be executed simultaneously to obtain the language description text features and the image description text features simultaneously.

[0068] S108, determining an item identifier corresponding to the item packaging based on the language description text features and the image description text features, and acquiring item identification data corresponding to the item identifier;

[0069] In one embodiment, after obtaining the language description text features and the image description text features of the article packaging in the image data, the information represented by the language description text features and the image description text features is integrated to obtain an article identification corresponding to the article packaging. The article identification may be an identification corresponding to the article packaged according to the article packaging, such as the name of the article, the brand name of the article, etc. The article database corresponding to the article type of the article packaging is searched according to the article identification to obtain the article identification data corresponding to the article identification.

[0070] Specifically, the item type may be a type of item that characterizes the item packaging, such as a medicine type, a food type, etc. The item type may be determined based on the item identification; the item database may be a database for storing data related to the item type, such as a medicine database or a food database; the item identification data may be identification data for introducing the item corresponding to the item packaging, such as instructions for use and precautions for the medicine.

[0071] In the embodiments of the present specification, by identifying the image data of the article packaging, the text data of the image data is obtained, the language description text features corresponding to the text data and the image description text features corresponding to the image data are extracted, and based on the language description text features and the image description text features, the user can take a photo of the article packaging through the terminal device, and obtain the article identification data corresponding to the article packaging according to the language description text features and the image description text features corresponding to the photographed image data, so as to achieve the effect of identifying the article packaging according to the combination of text and image, thereby improving the convenience and accuracy of object identification.

[0072] The following will be combined Figure 3 A specific example of an object identification method provided in an embodiment of this specification is described in detail.

[0073] S202, acquiring image data corresponding to the article packaging, and identifying text data in the image data;

[0074] In one embodiment, image data corresponding to article packaging is obtained. The article packaging may be packaging used to package and protect articles to facilitate transportation and sale of the articles, such as a packaging box or a packaging bag. The image data may be data obtained by photographing the article packaging. The article packaging in the image data is identified to obtain text data in the article packaging. The text data may be data obtained by identifying the image data according to a text recognition model.

[0075] Specifically, the method for acquiring image data may be: the user uses the camera component of the terminal device to photograph the packaging of the article to obtain data, and the packaging of the article in the image data may be any side of the packaging of the article. It is understandable that in order to improve the accuracy of identifying the packaging of the article, a feasible method is to photograph the front side of the packaging of the article to obtain data, wherein the front side of the packaging of the article may be one of the sides of the packaging of the article pre-set during the design and production process of the packaging of the article.

[0076] Exemplarily, when obtaining image data of an article packaging, the user can control the camera component of the terminal device to shoot the article packaging by running the "scan" function of the terminal device, thereby obtaining the image data corresponding to the article packaging; the user can also control the terminal device to access the image database stored in the terminal device, and select any image in the image database as image data. The image database can be a library used by the terminal device to store image data, for example, it can be a "gallery" folder of the terminal device. It can be understood that the image data can also be obtained by running other functions of the terminal device, and the method used to obtain the image data can be set according to the actual situation. It should be noted that if the image data includes more than one article, each article in each image data can be selected based on the attention mechanism to determine the article packaging in the image data. Therefore, the method used to obtain the image data can be a method with an attention mechanism to identify the image data.

[0077] For example, Figure 4 As shown, Figure 4 The image data shows a hand holding a medicine box. The objects in the image data include "a hand" and a "medicine box". After the image data is recognized according to the attention mechanism, the object packaging corresponding to the determined image data is a "medicine box".

[0078] The text data may be text data such as the name of the item included in the item packaging, or text data corresponding to the icon on the item packaging, obtained by extracting text from the image data using a text extraction model. The text extraction model may be a model with a text extraction function, such as OCR (Optical Character Recognition).

[0079] S204, feature encoding the text data to obtain language description text features corresponding to the text data;

[0080] In one embodiment, after acquiring the text data in the image data, the text data is encoded based on a text encoder to obtain language description text features that can be recognized by the model. The language description text features can be feature data that can be recognized by the model after the text data is formatted, so that the language description text features can be subsequently recognized to obtain the language description corresponding to the item packaging.

[0081] The encoder used to convert text data into language description text features may be a Tokenizer encoder, which converts text data of natural language text into language description text features in digital vector format.

[0082] S206, dividing the image data into at least one image block, performing feature extraction on each image block to obtain a visual feature, and determining an image association feature corresponding to each visual feature based on a position of each image block in the image data;

[0083] In one embodiment, after acquiring the image data, the image data is segmented based on a preset segmentation size to obtain at least one image block corresponding to the image data, and a feature encoder is used to extract features from each image block to obtain visual features of each image block. The feature encoder may be an encoder with visual feature compression and extraction, such as a ViT (Vision Transfoemer) encoder. Image association features corresponding to each visual feature are determined based on the position of each image block in the image data. The visual features may be features that characterize the user's visual perception, such as gradient features, color features, and texture features. The image association features may be features used to characterize the position of each visual feature in the image data, thereby identifying the visual features based on the image association features to improve the recognition accuracy of each visual feature.

[0084] Specifically, a feasible method for segmenting image data may be: obtaining the image size of the image data, scaling the image data based on a preset segmentation size and image size, and obtaining a scaled image corresponding to the image data, the image size may be a size representing the image data, specifically, may be based on the resolution of the image data, etc., the segmentation size may be a pre-set size for segmenting the image data, for example, may be 48*48, etc., and the specific value may be set according to actual conditions, the scaled image may be an image that is obtained by scaling the image data according to the image size so as to be easy to segment, and the scaled image is segmented based on the image size to obtain at least one image block corresponding to the image data.

[0085] For example, if the segmentation size is 48*48 and the image size of the image data is 480*480, then because the long side of the image size and the long side of the segmentation size, as well as the wide side of the image size and the wide side of the segmentation size are integer multiples, when the image data is segmented according to the segmentation size, the image data can be segmented completely, and the resulting image blocks do not overlap or are incomplete, so the image data does not need to be scaled; if the segmentation size is 48*48 and the image size of the image data is 640*480, then a feasible method is to scale the image data to a scaled image of 480*480. It is understandable that the scaling ratio when scaling the image data can be set according to actual conditions.

[0086] For example, Figure 5 As shown, Figure 5The image size of the image data is 640*480, and the segmentation size is 48*48, then the image data is scaled to a scaled image of 480*480, and the scaled image is segmented based on the segmentation size to obtain image blocks corresponding to the image data.

[0087] S208, performing feature mapping based on the image association features and each visual feature to obtain image description text features corresponding to the image data;

[0088] In one embodiment, after obtaining the visual features corresponding to the image data and the image association features corresponding to each visual feature, feature mapping is performed on the visual features based on the positions of the visual features represented by the image association features in the image data to obtain image description text features corresponding to the image data. The image description text features can be used to represent the description content corresponding to the image block. It can be understood that feature mapping of visual features based on image association features and identifying them in combination with the content in the visual features and the image association features can improve the accuracy of identifying visual features.

[0089] For example, Figure 6 As shown, Figure 6 The visual features of the image block are feature mapped to obtain the image description text features used to describe the image block, "an image block including an item trademark, the item trademark is a white elliptical outline, and the outline includes the word "YP"."

[0090] It should be noted that the execution order of obtaining the language description text features according to step S204 and obtaining the image description text features according to steps S206 and S208 can be set according to actual needs. Step S204 can be executed first to obtain the language description text features, and then steps S206 and S208 can be executed to obtain the image description text features; steps S206 and S208 can be executed first to obtain the image description text features, and then step S204 can be executed to obtain the language description text features; and step S204 can also be executed simultaneously to obtain the language description text features according to steps S206 and S208.

[0091] S210, inputting the language description text features and the image description text features into a recognition model, and obtaining an item identification corresponding to the item packaging based on the recognition model;

[0092] In one embodiment, after obtaining the language description text features and image description text features of the article packaging in the image data, the language description text features and the image description text features are input into a recognition model, and the recognition model is used to recognize the language description text features and the image description text features, so as to integrate and recognize the information represented by the language description text features and the image description text features, and obtain an article identification corresponding to the article packaging. The article identification can be an identification corresponding to the article packaged according to the article packaging, such as the name of the article, the brand name of the article, etc.

[0093] The recognition model may be pre-trained based on the item type of the item packaging and the sample data corresponding to the item type, and the recognition model may be an LLM model (Large Language Model). It is understandable that in order to improve the recognition accuracy of the recognition model, a feasible method may be to train the initial recognition model based on the item type to obtain a recognition model with an item type bias, for example, the initial recognition model may be trained based on the drug packaging as sample data to obtain a drug recognition model with the item type being a drug. Among them, the initial recognition model may be a large language model that has an item recognition function but is not biased towards the item type.

[0094] S212, searching in the item database based on the item identifier using the recognition model to obtain item recognition data corresponding to the item packaging;

[0095] In one embodiment, after obtaining the object identification corresponding to the image data, a recognition model is used to generate an acquisition request for object identification data based on the object identification, and the acquisition request is transmitted to an object database corresponding to the object type. The object data acquisition interface responds to the object identification data returned by the acquisition request, the object identification data.

[0096] Specifically, the item type may be a type of item that characterizes the item packaging, such as a medicine type, a food type, etc. The item type may be determined based on the item identification; the item database may be a database for storing data related to the item type, such as a medicine database or a food database; the item identification data may be identification data for introducing the item corresponding to the item packaging, such as instructions for use and precautions for the medicine.

[0097] The acquisition request may be a request for acquiring item identification data generated according to the item identifier, and the acquisition request may include a preset data type. For example, when the item type is a medicine, the data type in the acquisition request may be instructions for use and precautions for the medicine. After the item database acquires the acquisition request, the item database responds to the data type in the acquisition request, searches according to the item identifier, and obtains the item identification data corresponding to the acquisition request.

[0098] S214, using a recognition model to generate item introduction data corresponding to the item packaging based on the item identifier and the item recognition data;

[0099] In one embodiment, after using a recognition model to obtain the item identification and item identification data corresponding to the image data, the item identification and item identification data are substituted into a pre-set data template to obtain item introduction data corresponding to the item packaging. The item introduction data can be used to facilitate users to view the data included in the item packaging, as well as introduction data related to the item packaging, such as how to use it.

[0100] The data template may be a pre-set template for displaying data, so as to improve the convenience and viewing experience of users viewing item identification and item identification data. The specific style of the item template may be set according to actual conditions. Figure 7 As shown, Figure 7 The data template in is a drug data template for the item introduction data corresponding to the drug. The item identifier and item identification data are substituted into the data template to obtain the item introduction data.

[0101] S216, outputting item introduction data;

[0102] In one embodiment, after the item introduction data of the item packaging is adopted, the item introduction data is output and displayed on the display interface of the user's terminal device, so that after the user takes a picture of the item packaging through the terminal device, the terminal device can obtain the item introduction data output after identifying the item packaging, which is used to introduce the item corresponding to the item packaging, thereby improving the convenience of the user in querying the relevant information of the item corresponding to the item packaging.

[0103] It should be noted that if the item identification model is stored in the terminal device, the terminal device can control the item introduction data to be output and displayed on the display interface of the terminal device; if the item identification model is set in the server, the item identification model in the server generates the item introduction data and transmits the item introduction data to the terminal device, so that the terminal device controls the item introduction data to be output and displayed on the display interface of the terminal device.

[0104] Furthermore, if an inquiry statement regarding the packaging of the item is obtained, feature encoding is performed on the inquiry statement. The inquiry statement may be text data or audio data input by the user for inquiring about the packaging of the item. Inquiry features corresponding to the inquiry statement are obtained. The inquiry features may be a model language that can be recognized by a recognition model obtained after feature extraction of the inquiry statement, which characterizes the inquiry content of the inquiry statement. Response data is generated and output based on the inquiry features and the item identification data.

[0105] Specifically, the method for generating reply data based on the inquiry feature and the item identification data may be: the recognition model generates an inquiry request corresponding to the inquiry feature based on the inquiry feature, searches in the item database based on the inquiry request, obtains the answer data returned by the item database in response to the inquiry request, adds the answer data to a preset reply template, and generates reply data corresponding to the inquiry statement. The specific style of the reply template may be set according to actual conditions.

[0106] Among them, the response data can be data used to respond to the inquiry statement. For example, if the inquiry statement is "how to store medicines", the response data generated for the inquiry statement can be "place it in a cool area, avoid direct sunlight, and the recommended storage temperature is 10 to 25°C."

[0107] Furthermore, the query sentence input by the user may be a sentence input synchronously when the terminal device photographs the packaging of the item to obtain the image data; or it may be a sentence input by the user for the item introduction data after the terminal device outputs the item introduction data. It is understandable that if the user inputs the sentence synchronously when inputting the image data, the answer data may be added to the item introduction data after the item introduction data is generated, and the item introduction data to which the answer data is added may be output and displayed on the display interface of the terminal device.

[0108] It is understandable that if the user has preset that the item introduction data is not displayed when a query statement is obtained, the terminal device may only output the reply data when outputting the reply data without adding the reply data to the item introduction data.

[0109] In the embodiment of the present specification, by identifying the image data of the article packaging, the text data of the image data is obtained, the language description text features corresponding to the text data and the image description text features corresponding to the image data are extracted, and based on the language description text features and the image description text features, the user can shoot the article packaging through the terminal device, and obtain the article recognition data corresponding to the article packaging according to the language description text features and the image description text features corresponding to the photographed image data, so as to achieve the effect of recognizing the article packaging according to the combination of text and image, thereby improving the convenience and accuracy of article recognition. Further, the article introduction data is generated according to the article identification and the article recognition data layer, and the article introduction data is output, so as to facilitate the user to view the data introducing the article packaging in the image data, and improve the convenience of the user to identify the article. Further, according to the query sentence input by the user and the recognition result of the article packaging in the image data, the reply data corresponding to the query sentence is generated, so as to achieve the answer to the user's query sentence, facilitate the user to identify and understand the article corresponding to the article packaging, and improve the accuracy and convenience of article recognition.

[0110] based on Figure 1 The system architecture will be combined with Figure 8 and Fig. 9 , the object identification device provided in the embodiment of this specification is introduced in detail. It should be noted that, Figure 8 The object identification device 1 is used to execute the present invention Figure 2-Figure 7 For the convenience of explanation, only the part related to the embodiment of this specification is shown. For the specific technical details not disclosed, please refer to this specification. Figure 2-Figure 7 The embodiment shown.

[0111] A data acquisition unit 11 is used to acquire image data corresponding to the article packaging and identify text data in the image data;

[0112] A feature acquisition unit 12 is used to perform feature encoding on the text data to obtain language description text features corresponding to the text data;

[0113] The feature acquisition unit 12 is further used to acquire visual features of the image data, and generate image description text features of the image data based on the visual features;

[0114] The article identification unit 13 is used to determine the article identification corresponding to the article packaging based on the language description text feature and the image description text feature, and obtain article identification data corresponding to the article identification.

[0115] Optionally, the feature acquisition unit 12 is further used for:

[0116] The image data is divided into at least one image block, and features are extracted from each image block to obtain a visual feature, and image-related features corresponding to each visual feature are determined based on the position of each image block in the image data;

[0117] Feature mapping is performed based on the image association features and each of the visual features to obtain image description text features corresponding to the image data.

[0118] Optionally, the feature acquisition unit 12 is further used for:

[0119] Segmenting the image data based on a preset segmentation size to obtain at least one image block;

[0120] A feature encoder is used to extract features from each of the image blocks to obtain visual features corresponding to each of the image blocks.

[0121] Optionally, the feature acquisition unit 12 is further used for:

[0122] Acquiring an image size of the image data, scaling the image data based on a preset segmentation size and the image size, and obtaining a scaled image corresponding to the image data;

[0123] The scaled image is segmented based on the segmentation size to obtain at least one image block corresponding to the image data.

[0124] Optionally, the object identification unit 13 is further used for:

[0125] Inputting the language description text features and the image description text features into a recognition model, and obtaining an item identification corresponding to the item packaging based on the recognition model, wherein the recognition model is pre-trained based on the item type of the item packaging and sample data corresponding to the item type;

[0126] The identification model is used to search in an article database based on the article identifier to obtain article identification data corresponding to the article packaging.

[0127] Optional, such as Fig. 9 As shown, the object identification device 1 further includes a data output module 14, which is used to:

[0128] Generate the article introduction data corresponding to the article packaging by using the recognition model based on the article identification and the article identification data;

[0129] Output the item introduction data.

[0130] Optionally, the data output module 14 is further used for:

[0131] If a query statement for the article packaging is obtained, feature encoding is performed on the query statement to obtain a query feature corresponding to the query statement;

[0132] Reply data is generated and output based on the query characteristics and the item identification data.

[0133] In the embodiment of the present specification, by identifying the image data of the article packaging, the text data of the image data is obtained, the language description text features corresponding to the text data and the image description text features corresponding to the image data are extracted, and based on the language description text features and the image description text features, the user can shoot the article packaging through the terminal device, and obtain the article recognition data corresponding to the article packaging according to the language description text features and the image description text features corresponding to the photographed image data, so as to achieve the effect of recognizing the article packaging according to the combination of text and image, thereby improving the convenience and accuracy of article recognition. Further, the article introduction data is generated according to the article identification and the article recognition data layer, and the article introduction data is output, so as to facilitate the user to view the data introducing the article packaging in the image data, and improve the convenience of the user to identify the article. Further, according to the query sentence input by the user and the recognition result of the article packaging in the image data, the reply data corresponding to the query sentence is generated, so as to achieve the answer to the user's query sentence, facilitate the user to identify and understand the article corresponding to the article packaging, and improve the accuracy and convenience of article recognition.

[0134] The present specification also provides a computer storage medium that can store multiple program instructions, which are suitable for being loaded and executed by a processor as described above. Figure 1-Figure 7 The method steps of the embodiment shown in the figure can be found in the specific implementation process. Figure 1-Figure 7 The specific description of the illustrated embodiment will not be repeated here.

[0135] The embodiment of the present specification also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor as described above. Figure 1-Figure 7 The item recommendation model training method of the embodiment shown in the figure can be found in the specific execution process. Figure 1-Figure 7 The specific description of the illustrated embodiment will not be repeated here.

[0136] See also Fig.10 , is a schematic diagram of the structure of an electronic device provided in the embodiment of this specification. Fig.10As shown, the electronic device 1000 may include: at least one processor 1001, such as a CPU, at least one network interface 1004, an input / output interface 1003, a memory 1005, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1005 may optionally be at least one storage device located away from the aforementioned processor 1001. As shown in FIG. Fig.10 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, an input and output interface module, and an object identification application.

[0137] exist Fig.10 In the electronic device 1000 shown, the input-output interface 1003 is mainly used to provide an input interface for the user and obtain data input by the user.

[0138] In one embodiment, the processor 1001 may be used to call the object identification application stored in the memory 1005, and specifically perform the following operations:

[0139] Acquire image data corresponding to the article packaging, and identify text data in the image data;

[0140] Performing feature encoding on the text data to obtain language description text features corresponding to the text data;

[0141] Acquire visual features of the image data, and generate image description text features of the image data based on the visual features;

[0142] An article identifier corresponding to the article packaging is determined based on the language description text feature and the image description text feature, and article identification data corresponding to the article identifier is acquired.

[0143] Optionally, when the processor 1001 acquires the visual features of the image data and generates the image description text features of the image data based on the visual features, the processor 1001 specifically performs the following operations:

[0144] The image data is divided into at least one image block, and features are extracted from each image block to obtain a visual feature, and image-related features corresponding to each visual feature are determined based on the position of each image block in the image data;

[0145] Feature mapping is performed based on the image association features and each of the visual features to obtain image description text features corresponding to the image data.

[0146] Optionally, when the processor 1001 divides the image data into at least one image block and extracts features from each image block to obtain visual features, the processor 1001 specifically executes:

[0147] Segmenting the image data based on a preset segmentation size to obtain at least one image block;

[0148] A feature encoder is used to extract features from each of the image blocks to obtain visual features corresponding to each of the image blocks.

[0149] Optionally, when the processor 1001 divides the image data based on a preset dividing size to obtain at least one image block, the processor 1001 specifically performs:

[0150] Acquiring an image size of the image data, scaling the image data based on a preset segmentation size and the image size, and obtaining a scaled image corresponding to the image data;

[0151] The scaled image is segmented based on the segmentation size to obtain at least one image block corresponding to the image data.

[0152] Optionally, when the processor 1001 determines the item identifier corresponding to the item packaging based on the language description text feature and the image description text feature and obtains the item identification data corresponding to the item identifier, the processor 1001 specifically performs the following operations:

[0153] Inputting the language description text features and the image description text features into a recognition model, and obtaining an item identification corresponding to the item packaging based on the recognition model, wherein the recognition model is pre-trained based on the item type of the item packaging and sample data corresponding to the item type;

[0154] The identification model is used to search in an article database based on the article identifier to obtain article identification data corresponding to the article packaging.

[0155] Optionally, after acquiring the item identification data corresponding to the item identifier, the processor 1001 further performs the following operations:

[0156] Generate the article introduction data corresponding to the article packaging by using the recognition model based on the article identification and the article identification data;

[0157] Output the item introduction data.

[0158] Optionally, the processor 1001 further performs the following operations:

[0159] If a query statement for the article packaging is obtained, feature encoding is performed on the query statement to obtain a query feature corresponding to the query statement;

[0160] Reply data is generated and output based on the query characteristics and the item identification data.

[0161] In the embodiment of the present specification, by identifying the image data of the article packaging, the text data of the image data is obtained, the language description text features corresponding to the text data and the image description text features corresponding to the image data are extracted, and based on the language description text features and the image description text features, the user can shoot the article packaging through the terminal device, and obtain the article recognition data corresponding to the article packaging according to the language description text features and the image description text features corresponding to the photographed image data, so as to achieve the effect of recognizing the article packaging according to the combination of text and image, thereby improving the convenience and accuracy of article recognition. Further, the article introduction data is generated according to the article identification and the article recognition data layer, and the article introduction data is output, so as to facilitate the user to view the data introducing the article packaging in the image data, and improve the convenience of the user to identify the article. Further, according to the query sentence input by the user and the recognition result of the article packaging in the image data, the reply data corresponding to the query sentence is generated, so as to achieve the answer to the user's query sentence, facilitate the user to identify and understand the article corresponding to the article packaging, and improve the accuracy and convenience of article recognition.

[0162] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0163] The above disclosure is only the preferred embodiment of this specification, which certainly cannot be used to limit the scope of rights of this specification. Therefore, equivalent changes made according to the claims of this specification are still within the scope covered by this specification.

Claims

1. A method for identifying an object, the method comprising: Acquire image data corresponding to the article packaging, and identify text data in the image data; Performing feature encoding on the text data to obtain language description text features corresponding to the text data; Acquire visual features of the image data, and generate image description text features of the image data based on the visual features; An article identifier corresponding to the article packaging is determined based on the language description text feature and the image description text feature, and article identification data corresponding to the article identifier is acquired.

2. According to the method of claim 1, the step of acquiring visual features of the image data and generating image description text features of the image data based on the visual features comprises: Dividing the image data into at least one image block, performing feature extraction on each image block to obtain a visual feature, and determining an image association feature corresponding to each visual feature based on a position of each image block in the image data; Feature mapping is performed based on the image association features and each of the visual features to obtain image description text features corresponding to the image data.

3. The method according to claim 2, wherein dividing the image data into at least one image block and extracting features from each image block to obtain visual features comprises: Segmenting the image data based on a preset segmentation size to obtain at least one image block; A feature encoder is used to extract features from each of the image blocks to obtain visual features corresponding to each of the image blocks.

4. The method according to claim 3, wherein the step of dividing the image data based on a preset dividing size to obtain at least one image block comprises: Acquiring an image size of the image data, scaling the image data based on a preset segmentation size and the image size, and obtaining a scaled image corresponding to the image data; The scaled image is segmented based on the segmentation size to obtain at least one image block corresponding to the image data.

5. The method according to claim 1, wherein determining the item identification corresponding to the item packaging based on the language description text feature and the image description text feature, and obtaining the item identification data corresponding to the item identification comprises: Inputting the language description text features and the image description text features into a recognition model, and obtaining an item identification corresponding to the item packaging based on the recognition model, wherein the recognition model is pre-trained based on the item type of the item packaging and sample data corresponding to the item type; The identification model is used to search in an article database based on the article identifier to obtain article identification data corresponding to the article packaging.

6. The method according to claim 1, after obtaining the item identification data corresponding to the item identifier, further comprising: Generate the article introduction data corresponding to the article packaging by using the recognition model based on the article identification and the article identification data; Output the item introduction data.

7. The method according to claim 1, further comprising: If a query statement for the article packaging is obtained, feature encoding is performed on the query statement to obtain a query feature corresponding to the query statement; Reply data is generated and output based on the query characteristics and the item identification data.

8. An object identification device, comprising: A data acquisition unit, used to acquire image data corresponding to the article packaging, and identify text data in the image data; A feature acquisition unit, used for performing feature encoding on the text data to obtain language description text features corresponding to the text data; The feature acquisition unit is further used to acquire visual features of the image data, and generate image description text features of the image data based on the visual features; The article identification unit is used to determine the article identification corresponding to the article packaging based on the language description text feature and the image description text feature, and obtain the article identification data corresponding to the article identification.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program code, and when the computer program code is executed, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device comprising: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the method as claimed in any one of claims 1 to 7.

11. A computer program product having at least one instruction stored thereon, wherein when the at least one instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images