Object recognition method and electronic device

Abstract

The embodiment of the present application discloses an object recognition method and an electronic device, the method comprising: determining a target object group to be recognized, the target object group including at least two target objects to be recognized, the target objects being associated with text information and image information; inputting the text information and image information of each target object into a target model for recognition processing; the target model is used to fuse the text features and image features of each target object respectively, and judge whether each target object has commonality in target attributes based on the obtained text-image cross-features; the target model includes a multimodal feature fusion model, which is used to fuse the text features and image features in the process of performing the task of judging whether the text features and image features match. Through the embodiment of the present application, it is possible to more accurately identify whether the target objects have commonality in target attributes.

Description

Technical Field

[0001] The present application relates to the field of information recognition technology, and in particular to an object recognition method and an electronic device. Background Art

[0002] In the commodity object information system, there is a need to identify the same or similar commodities in many application scenarios. For example, when recommending commodities to users, it is necessary to identify commodities similar to the commodities that the user has browsed in the past from the commodity library for recommendation. Alternatively, in some pages that aggregate and display multiple commodities (for example, event venue pages, etc.), it may be necessary to break up the same or similar commodities before displaying them to avoid excessive aggregation of the same or similar commodities. At this time, it is also necessary to identify the same or similar commodities from the set of commodities to be displayed, and then break them up according to the recognition results, and so on.

[0003] When identifying identical or similar products, the solutions in the prior art are usually based on similarity comparisons of product images or texts. That is, by comparing the product images of two products, it can be determined whether the two products are identical or similar; or by comparing the text information such as the titles of the two products, it can be determined whether the two products are identical or similar. However, the image may not be the main image or the main subject may not be obvious, and the text may also have similar inaccurate attribute information (for example, when a merchant publishes a product, there may be problems such as missing or incorrect product text information, redundant information accumulation, and lack of key information), which ultimately leads to deviations in the judgment results of identical or similar products.

[0004] Therefore, how to improve the accuracy of identifying the same / similar products has become a technical problem that needs to be solved by those skilled in the art. Summary of the invention

[0005] The present application provides an object recognition method and an electronic device, which can more accurately identify whether the target objects have commonalities in target attributes. In the commodity object scenario, the accuracy of identifying the same / similar commodities can be improved.

[0006] This application provides the following solutions:

[0007] An object recognition method, comprising:

[0008] Determine a target object group to be identified, wherein the target object group includes at least two target objects to be identified, and the target objects are associated with text information and image information;

[0009] Inputting the text information and image information of each target object in the target object group into the target model for recognition processing to obtain an object recognition result; wherein the target model is used to fuse the text features and image features of each target object respectively, and judging whether each target object has commonality in target attributes based on the obtained text-image cross features of each target object;

[0010] The target model includes a multimodal feature fusion model, which is used to fuse the text features and image features to obtain the image-text cross-features during the process of performing the task of judging whether the text features and image features match.

[0011] Wherein, the target model includes a feature generation model and a discrimination model, and the feature generation model includes a feature extraction model and a multimodal feature fusion model;

[0012] The feature extraction model is used to: extract text features and image features from each target object respectively;

[0013] The multimodal feature fusion model is used to: fuse text features with image features and output image-text cross features of each target object;

[0014] The discriminant model is used to determine whether each target object has commonality in target attributes according to the image-text cross-features of each target object.

[0015] The target model is a complete set of models formed by combining a feature generation model and a discrimination model.

[0016] Before training the target model, the method further includes:

[0017] Training the feature generation model, and obtaining parameter learning results in the feature generation model according to the training results;

[0018] The parameter learning results are used as initial parameter values ​​when training the target model.

[0019] Among them, it also includes:

[0020] Provide a recognition result on whether the target object has commonality in the target attribute, receive user feedback information on the accuracy of the recognition result, and iteratively train the target model according to the user feedback information.

[0021] Wherein, the target object includes a commodity object;

[0022] The determining whether the objects have commonality in the target attribute includes:

[0023] Determine whether each product object in the same product object group is a product object of the same or similar type.

[0024] Wherein, at least two commodity objects included in the commodity object group correspond to the same category;

[0025] The target model includes a plurality of different target models corresponding to different categories;

[0026] The step of inputting the text information and image information of each target object in the target object group into the target model for processing includes:

[0027] According to the target category to which each commodity object in the target commodity object group belongs, the text information and image information of each commodity object are input into the target model corresponding to the target category for processing.

[0028] Wherein, determining the target object group to be identified includes:

[0029] When recommending commodity objects based on commodity objects that the user has browsed historically, the commodity objects that have been browsed historically and other data objects in the commodity object library are combined into a plurality of target commodity object groups.

[0030] Wherein, determining the target object group to be identified includes:

[0031] After receiving a request submitted by a user to search for a product object of the same item or similar item as a target product object, the target product object and other data objects in the product object library are combined into a plurality of target product object groups.

[0032] Wherein, determining the target object group to be identified includes:

[0033] Receiving text information and image information of an object to be identified submitted by a user, wherein the image information is generated by capturing an image of a physical object corresponding to the object to be identified;

[0034] The object to be identified and a plurality of commodity objects in a commodity object library are respectively formed into a target object group; the commodity objects in the commodity object library are also associated with tags, and the tags are used to indicate whether the corresponding commodity objects are genuine or how close they are to the genuine ones;

[0035] The method further comprises:

[0036] According to the label corresponding to the target commodity object whose similarity with the object to be identified meets the conditions, identification result information on whether the object to be identified is an authentic product or the degree of similarity to the authentic product is provided.

[0037] A model processing method, comprising:

[0038] Constructing an image-text matching model, wherein the image-text matching model is used to identify whether the image and the text are used to describe the same object;

[0039] Acquire a training sample, wherein the training sample includes a plurality of sample entries, wherein the sample entries include image content and text content, and annotation information regarding whether the image content and the text content match;

[0040] Training the image-text matching model using the training samples;

[0041] After the training is completed, the multimodal feature fusion model is generated according to the feature fusion module in the image-text matching model to extract image features and text features of the input target object, and to fuse the image text with the text features to generate image-text cross features.

[0042] A model processing method, wherein the model is formed by combining a feature generation model and a discriminant model, and the method comprises:

[0043] The feature generation model is trained to determine parameter values ​​therein; the feature generation model is used to fuse the image features and text features of the target object to generate image-text cross features;

[0044] Acquire a training sample, wherein the training sample includes a plurality of target object groups, wherein the target object group includes at least two target objects, and annotation information regarding whether the at least two target objects have commonality in target attributes;

[0045] The target model is trained using the training samples, wherein parameter values ​​determined during the training of the feature generation model are used as initial values ​​of the parameters when training the target model.

[0046] A method for providing a recognition result, comprising:

[0047] Receiving text information and image information of an object to be identified submitted by a user, wherein the image information is generated by capturing an image of a physical object corresponding to the object to be identified;

[0048] The object to be identified and a plurality of commodity objects in a commodity object library are respectively formed into a target object group; the commodity objects in the commodity object library are also associated with tags, and the tags are used to indicate whether the corresponding commodity objects are genuine or how close they are to the genuine ones;

[0049] Inputting the text information and image information of each target object in the target object group into a target model for processing, the target model is used to fuse the text features and image features of each target object respectively, and determine the similarity between the target objects according to the obtained image-text cross features; wherein the target model includes a multimodal feature fusion model, and the multimodal feature fusion model is used to fuse the text features and image features in the process of performing the task of judging whether the text features and the image features match, so as to obtain the image-text cross features;

[0050] According to the label corresponding to the target commodity object whose similarity with the object to be identified meets the conditions, identification result information on whether the object to be identified is an authentic product or the degree of similarity to the authentic product is provided.

[0051] An object recognition device, comprising:

[0052] A target object group determination unit, used to determine a target object group to be identified, wherein the target object group includes at least two target objects to be identified, and the target objects are associated with text information and image information;

[0053] The recognition processing unit is used to input the text information and image information of each target object in the target object group into the target model for recognition processing to obtain an object recognition result; wherein the target model is used to fuse the text features and image features of each target object respectively, and judge whether each target object has commonality in target attributes based on the obtained text-image cross features of each target object;

[0054] The target model includes a multimodal feature fusion model, which is used to fuse the text features and image features to obtain the image-text cross-features during the process of performing the task of judging whether the text features and image features match.

[0055] A model processing device, comprising:

[0056] A model building unit, used to build an image-text matching model, wherein the image-text matching model is used to identify whether an image and a text are used to describe a unified object;

[0057] A training sample acquisition unit, used to acquire a training sample, wherein the training sample includes a plurality of sample entries, wherein the sample entries include image content and text content, and annotation information on whether the image content matches the text content;

[0058] A training unit, used for training the image-text matching model through the training samples;

[0059] The multimodal feature fusion model generation unit is used to generate the multimodal feature fusion model after the training is completed, according to the feature fusion module in the image-text matching model, so as to extract image features and text features of the input target object, and fuse the image text with the text features to generate image-text cross features.

[0060] A model processing device, wherein the model is formed by combining a feature generation model and a discriminant model, and the device comprises:

[0061] The first training unit is used to train the feature generation model and determine the parameter values ​​therein; the feature generation model is used to fuse the image features and text features of the target object to generate image-text cross features;

[0062] A training sample acquisition unit, used to acquire training samples, wherein the training samples include a plurality of target object groups, each target object group includes at least two target objects, and annotation information about whether the at least two target objects have commonality in target attributes, wherein each commodity object is associated with text information and image information;

[0063] The second training unit is used to train the target model using the training samples, wherein the parameter values ​​determined during the training of the feature generation model are used as initial values ​​of the parameters when training the target model.

[0064] A device for providing recognition results, comprising:

[0065] An information receiving unit, used to receive text information and image information of an object to be identified submitted by a user, wherein the image information is generated by capturing an image of a physical object corresponding to the object to be identified;

[0066] a target object group generating unit, used to form target object groups with the to-be-identified object and a plurality of commodity objects in a commodity object library; the commodity objects in the commodity object library are also associated with tags, and the tags are used to indicate whether the corresponding commodity objects are genuine or how close they are to the genuine ones;

[0067] A similarity recognition unit is used to input the text information and image information of each target object in the target object group into a target model for processing, wherein the target model is used to fuse the text features and image features of each target object respectively, and determine the similarity between the target objects according to the obtained image-text cross feature; wherein the target model includes a multimodal feature fusion model, and the multimodal feature fusion model is used to fuse the text features and image features in the process of performing the task of judging whether the text features and image features match, so as to obtain the image-text cross feature;

[0068] The recognition result providing unit is used to provide recognition result information on whether the object to be recognized is authentic or how close it is to the authentic product based on the label corresponding to the target commodity object whose similarity with the object to be recognized meets the conditions.

[0069] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any of the methods described above.

[0070] An electronic device, comprising:

[0071] one or more processors; and

[0072] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of any of the methods described above.

[0073] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0074] Through the embodiment of the present application, the target object's multimodal features (for example, including text information and image information, etc.) can be fused to obtain the target object's cross-image features, and then the target object can be more accurately identified based on the cross-image features whether there is commonality in the target attribute. Wherein, when fusing the multimodal features, considering that the image features and the text features belong to different domains, the image features and the text features are not directly spliced, or simple mathematical operations are performed, but a multimodal feature fusion model can be constructed, and the model can be trained in a supervised learning manner, so that the multimodal feature fusion model can be used to fuse the text features and the image features. In order to realize the supervised learning training of the multimodal feature fusion model, it can also be converted into the training of the image-text matching classification model, that is, the specific multimodal feature fusion model can be in the process of performing the task of judging whether the text features and the image features match, the text features and the image features are fused, and then the fusion processing result can be determined as the cross-image feature. In this way, a supervised multimodal feature fusion model can be generated to generate graphic and text cross-features that can better express the characteristics of the target object, and then more accurately identify whether the target objects have commonalities in target attributes (for example, whether the product objects belong to the same / similar models).

[0075] The specific discriminant model can also be obtained by training in a supervised learning manner. In addition, the feature generation model including the multimodal feature fusion model and the discriminant model can be spliced ​​together to form a set of target models, so as to realize the simultaneous training of the feature generation model and the discriminant model to improve the recognition accuracy of the model.

[0076] In addition, before training the target model composed of the feature generation model and the discriminant model, the feature generation model can be trained first. After determining the parameter values ​​in the feature generation model, they can be retained. Thereafter, the target model can be retrained, and the parameter values ​​determined when the feature generation model was trained alone can be used as the initial values ​​of the parameters in the model when retraining the target model. In this way, the accuracy of model recognition can be further improved.

[0077] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0079] Figure 1 It is a schematic diagram of the system architecture provided by the embodiment of the present application;

[0080] Figure 2 is a flow chart of the first method provided in an embodiment of the present application;

[0081] Figure 3 It is a schematic diagram of the multimodal fusion model training process provided in an embodiment of the present application;

[0082] Figure 4 is a schematic diagram of another model training process provided in an embodiment of the present application;

[0083] Figure 5 is a flow chart of the second method provided in an embodiment of the present application;

[0084] Figure 6 is a flowchart of the third method provided in an embodiment of the present application;

[0085] Figure 7 is a flowchart of the fourth method provided in an embodiment of the present application;

[0086] Figure 8 is a schematic diagram of a first device provided in an embodiment of the present application;

[0087] Fig. 9 is a schematic diagram of a second device provided in an embodiment of the present application;

[0088] Fig.10 is a schematic diagram of a third device provided in an embodiment of the present application;

[0089] Fig.11 is a schematic diagram of a fourth device provided in an embodiment of the present application;

[0090] Fig.12 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0091] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0092] In the embodiment of the present application, in order to improve the accuracy of the same / similar commodity object recognition, a recognition method of multimodal feature fusion is adopted. Specifically, since commodity objects generally include elements such as text (including titles, brand names, etc.) and images (pictures or videos, etc.), although there may be recognition deviations when commodity objects are recognized simply from the text feature or image feature dimension, if the text feature and the image feature are fused together for recognition, it can play a role in compensating for each other's shortcomings. Therefore, in the embodiment of the present application, the features of the single mode such as text features and image features are fused together to generate multimodal graphic cross features, and then recognition is performed based on the multimodal features, so as to improve the recognition accuracy of the same / similar commodity objects. However, in the specific implementation, how to fuse the text features and the image features, and how to better identify the same / similar commodity objects according to the fused graphic cross features, are issues that need to be considered.

[0093] Among them, regarding the problem of multimodal feature fusion, in the prior art, one solution is to directly splice the feature vectors of multiple different modes together to generate multimodal features. For example, the image feature vector is (1, 2, 3) and the text vector is (56, 67, 90, 4), then the fused feature can be (1, 2, 3, 56, 67, 90, 4). Alternatively, in some solutions, the fused feature can be obtained after performing some mathematical operations on the image features and text features. For example, for the above-mentioned image feature vector and text feature vector, the fused multimodal feature vector can be (1*56, 1*67, 1*90, 1*4, 2*56, 2*67, 2*90, 2*4, 3*56, 3*67, 3*90, 3*4), and so on.

[0094] However, the inventors of the present application discovered in the process of implementing the present application that, since text features and image features belong to features of different domains, if the two are directly spliced ​​together or simple mathematical operations are performed on the two, the fused features may still be insufficient to express the features of the commodity object. Therefore, in the embodiments of the present application, the fusion method of text features and image features is firstly improved.

[0095] Specifically, in order to more reasonably fuse text features and image features, a multimodal feature fusion model can be established through supervised learning. Then, through this multimodal feature fusion model, the text features and image features of the same product object can be fused into a more advanced abstract feature. The abstract feature can then be used as a cross-image feature in the subsequent recognition process.

[0096] In the process of supervised learning model training, since the samples need to be annotated in advance (usually by manual annotation), the training target (that is, the information expected to be output by the model) needs to be known before annotation. If the direct output result of a model is a feature vector, it is difficult to manually annotate the sample (it is also for this reason that similar feature generation models in the prior art are trained in an unsupervised manner). For this reason, in an embodiment of the present application, a model for performing a task of judging whether the text features and image features of a commodity object match (that is, judging whether an input text and an image can be used to describe the same commodity object) can be constructed first. Since in the process of training the model so that it can judge whether the text features and image features match, what needs to be learned is how to fuse the text features and image characteristics into a more advanced feature vector, that is, in the process of performing the task of judging whether the text features and image features match, the output result of one layer will include the feature vector after the text features and image features are fused, therefore, the vector can be used as the fused text-text cross vector in the embodiment of the present application to participate in the subsequent recognition process of the same / similar commodity objects.

[0097] In other words, in the embodiment of the present application, the training of the multimodal feature fusion model is converted into the training of the image-text matching classification model. In this way, a model for image-text matching classification can be constructed, for example, it can be specifically constructed based on the Bert model, etc. Then, the image-text matching classification model can be trained in a supervised manner. Specifically, multiple <text, image> pairs and corresponding annotation information can be used as training samples, wherein the annotation information can be whether the text matches the image. After the training is completed, the role of the model is to give a judgment result on whether the input text and image match. Of course, since it is not actually necessary to classify the matching of the image and the text in the embodiment of the present application, but only to obtain the fusion result of the text feature and the image feature, therefore, from the above-mentioned trained model, a layer (that is, the output layer) that is finally used to output the image-text matching judgment result can be removed from the model, so that after the text and image of the commodity object are input, the image-text cross feature vector after the text feature and the image feature are fused can be input.

[0098] Through the above method, it is possible to fuse text features with image features based on supervised learning, rather than simply concatenating two feature vectors. Therefore, a more advanced abstract expression of product objects can be achieved. The cross-text and image features obtained by this fusion processing can express the characteristics of product objects more comprehensively and accurately.

[0099] After obtaining the graphic and text cross-features of the commodity object, there are multiple ways to identify the same / similar commodity objects based on the graphic and text cross-features. In a preferred embodiment of the present application, the specific discriminant model can also be obtained by training in a supervised learning manner. In addition, the feature generation model and the discriminant model can be spliced ​​together to form a target model, thereby realizing the simultaneous training of the feature generation model and the discriminant model.

[0100] In addition, in order to further improve the recognition accuracy, before training the target model composed of the feature generation model and the discriminant model, the feature generation model can be trained first, and after the parameter values ​​in the feature generation model are determined, they can be retained; thereafter, the target model can be retrained, and the parameter values ​​determined when the feature generation model was trained alone can be used as the initial values ​​of the parameters in the model when retraining the target model. In this way, the accuracy of model recognition can be further improved.

[0101] From the perspective of system architecture, the embodiment of the present application provides a function for identifying whether the target objects have commonalities in target attributes. This function can be provided to users as an independent application, or can also exist as a functional module of other systems. Figure 1 As shown, the object recognition function module can be provided in the commodity object information system, so that when other function modules in the commodity object information system need to determine whether certain commodity objects belong to the same or similar items, they can be identified through the object recognition model. For example, when the recommendation module needs to recommend commodity objects similar to the historically browsed commodities to the user, the function module can be used to obtain the same / similar commodity objects from the commodity pool. Alternatively, when the commodity objects in a certain page need to be scattered by style, the function module can also be used to identify the same / similar commodity objects and then scatter them. Alternatively, some commodity object information systems may also provide users with function modules such as "find similar" or "find the same item". For example, in a certain commodity object list page, in the process of displaying multiple commodity objects through multiple resource positions, operation controls such as "find similar" can also be provided in specific resource positions. After the user clicks the operation control, the commodity object can be identified with the commodity objects in the commodity library for the same / similar items, and the identification result is returned, and so on. At this time, the above-mentioned "find similar" or "find same style" and other functional modules can also be used to determine the specific group of commodity objects that need to be identified, and by calling the above-mentioned object identification functional module, it can be identified whether each commodity object in the group belongs to the same or similar style, etc.

[0102] Alternatively, the embodiment of the present application can also use the specific same / similar commodity object recognition capabilities to provide users with other updated forms of services. For example, in actual applications, a user may purchase a commodity through a certain channel, and may need to determine whether the commodity is genuine; or, a user A and a user B respectively purchase the same brand of cosmetics of the same item. At this time, it may be necessary to compare the two commodities to see which one is more like a genuine product, and so on. Therefore, the above-mentioned services can be provided to users. Specifically, relevant functional modules (or light applications, applets, etc.) can be provided in the commodity object information system, or independent applications can be provided, or handheld hardware devices can be provided, and so on. Through this service, users can use the purchased commodities (corresponding to the physical objects) as objects to be identified, and collect image information by taking pictures of them, and can upload specific image information. Based on the image information, the service can perform similarity recognition with the relevant commodities in the commodity library. In addition, the commodities in the commodity library can be pre-added with labels indicating whether they are genuine or the degree of proximity to the genuine products, and so on. In this way, if it is recognized that the current object to be identified has a high similarity with a certain commodity object in the commodity library, the label corresponding to the commodity object can be used to return the identification result of whether the object to be identified is authentic or how close it is to the authentic. For example, "your product may be authentic", etc. Alternatively, when uploading photos of two objects to be identified for authenticity comparison, the two objects to be identified can be identified for similarity with each related commodity in the commodity library, and then, based on the labels corresponding to the commodities with the highest similarity, the identification result of which of the two objects to be identified is more authentic can be given, etc.

[0103] It should be noted here that when the same or similar items are identified based on the image information uploaded by the user and the commodity objects in the commodity library, there may be no text content in the image information uploaded by the user. At this time, the image features in the image information can be directly abstracted to a higher level. Alternatively, text can be identified from the image, for example, including trademark text, etc., and then text features can be extracted. Alternatively, an input control for inputting specific text content can be provided to the user, so that the user can enter text content for the specific object to be identified to improve the recognition accuracy, etc.

[0104] It should also be noted that the solution provided in the embodiment of the present application is not limited to the use in the scene of identifying the same or similar commodity objects, but can also be extended to other fields. For example, it can provide users with the ability to summarize pictures (for example, it can summarize similar pictures in the mobile phone album), or it can also be used to distinguish between original images and images processed by PS, etc.

[0105] The specific implementation scheme provided in the embodiments of the present application is described in detail below.

[0106] Embodiment 1

[0107] First, this embodiment provides an object recognition method, see Figure 2 , the method may include:

[0108] S201: Determine a target object group to be identified, wherein the target object group includes at least two target objects to be identified, and the target objects are associated with text information and image information.

[0109] Among them, the target object group to be identified can be determined according to the actual identification needs. For example, in the scenario of the commodity object information system, as mentioned above, if it is necessary to recommend the same / similar commodity objects based on the user's historical browsing of commodities, the historically browsed commodities can be combined with each commodity in the commodity pool to be recommended to obtain multiple commodity pairs, and each commodity pair can become a commodity object group. Alternatively, when performing "find similar" according to a commodity object A currently displayed on the page, the commodity object A and the commodity objects in the commodity library can be combined into multiple commodity object groups (each commodity object group includes the commodity object A and another commodity object in the commodity library). In addition, in scenarios such as judging the authenticity of products based on photos uploaded by users, the photographed objects in the photos can be combined with the commodity objects in the commodity library into multiple commodity object groups for identification, etc.

[0110] After the target object group is determined, text information and image information can be collected for each target object. If the specific target objects are all commodity objects published in the system, the commodity information library can store the text information and image information associated with the specific commodity objects, wherein the text information can be obtained through the title, trademark and other information of the commodity object, and the image information can be obtained through the main image of the commodity object. If the specific target objects include objects to be identified specified by the user by uploading image information, text information and other methods, the text information and image information can be directly extracted from the information uploaded by the user. If the user only uploads image information, there may be a situation where the image information does not contain text information. Therefore, an alarm function can also be provided, that is, in the process of reading the text, image and other contents of each object to be identified, if a part of the content is found to be missing, an alarm can be triggered to remind the user whether the information is read incorrectly or ignored and continued to be read, or an operation control for inputting text information can also be provided for the user to input the associated text content for the specific uploaded image, and so on.

[0111] S202: Inputting the text information and image information of each object in the target object group into the target model for processing to obtain an object recognition result.

[0112] After determining the target object group, as well as the text content and image content associated with the specific target object, the pre-trained target model can be used to judge whether the target objects in the group have commonalities in target attributes, for example, whether the product objects in the same group are of the same or similar models, whether certain photos belong to the same scene, and so on.

[0113] Among them, the target model can be used to fuse the text features and image features of each target object respectively, and judge whether each target object has commonality in the target attribute according to the obtained text-image cross-features of each target object. In a specific implementation mode, the specific target model can include a feature generation model and a discriminant model. The specific feature generation model is used to: extract text features and image features of each target object respectively, fuse the text features with the image features, and output the text-image cross-features of each target object; the discriminant model is used to judge whether each target object has commonality in the target attribute according to the text-image cross-features of each target object. As for the specific feature generation model, it can be composed of a text feature extraction model, an image feature extraction model, and a multimodal feature fusion model. The text feature extraction model is mainly used to extract text features from the text content of the target object, and the image feature extraction model is used to extract image features from the image content of the target object, and obtain text feature vectors and image feature vectors respectively. The multimodal feature fusion model is used to fuse the text feature vector and the image feature vector to obtain the text-image cross-feature vector.

[0114] Among them, as mentioned above, in the embodiment of the present application, the multimodal feature fusion model can be generated by training in a supervised learning manner. In order to train the multimodal feature fusion model in a supervised learning manner to improve the quality of the feature fusion results, the training of the multimodal feature fusion model can be converted into the training of a model that classifies whether the image and text of the target object match. In other words, the specific multimodal feature fusion model fuses the text features and image features in the process of performing the task of judging whether the text features and image features of the target object match, and determines the fusion processing result as the image-text cross feature.

[0115] In this way, when training the multimodal feature model, a picture-text matching classification model can be established in advance to train the picture-text matching classification model. After completing the training of the picture-text matching classification model, the output layer in the model is removed to obtain the multimodal feature model.

[0116] Specifically, when training the image-text matching classification model, multiple training samples and corresponding annotation information can be obtained. Each training sample may correspond to an (image, text) pair, and the annotation information is whether the specific image and text match. For example, in the commodity object scenario, in the specific implementation, the main image and the corresponding title and other texts can be extracted in advance based on the multiple commodity objects that have been published in the commodity object information system as training samples, and the image and text are marked by manual annotation and other methods. Of course, if the image and text are of the same commodity object that have been published in the system, the two are usually matched. However, in the specific training, some negative samples are usually required, that is, samples where the image and text do not match. For this reason, in the specific implementation, such as Figure 3 As shown, it is also possible to obtain a pre-annotated data set D (including multiple images and corresponding texts, and the images and texts in most barcodes are matched), and then obtain a data set G by sampling each text and image in the data set D several times. For example, the original data set D includes (image 1, text 1), (image 2, text 2), (image 3, text 3), and so on. After several samplings, the data set G can include not only the above data items, but also (image 1, text 2), (image 1, text 3), (image 2, text 1), (image 2, text 3), (image 3, text 1), (image 3, text 2), and so on. In this way, not only a larger number of samples can be constructed, but also the samples can include not only positive samples but also negative samples. For example, the data set G has a total of n samples, and the sampled image-text matching samples are marked as 1, and the sampled image-text mismatch samples are marked as 0, and so on.

[0117] After obtaining the above data set G, each sample can be extracted by text encoding and Resnet and other algorithms to obtain text feature sequences and image feature sequences. For example, after performing text segmentation on the input text content and removing stop words, the text is encoded using predefined segmentation and other algorithms to extract text feature sequences. In addition, the input image can be cut into N small blocks (N can be set by itself, for example, it can be set to 64), and each small image uses predefined Resnet and other algorithms to extract features to obtain image feature sequences. Afterwards, algorithms such as Bert can be used to achieve the fusion of text feature sequences and image feature sequences, and Relu (Rectified Linear Unit, linear rectifier function) can be used as an activation function (that is, a function that specifically performs fusion calculations, mainly used to calculate the data in the text feature sequence and the image feature sequence to obtain a mapping result, that is, a fusion result). The obtained fusion result can be used to judge whether the text matches the image, and the parameter W1 in the model is solved by bringing in the loss function and using algorithms such as gradient descent. After multiple rounds of iterations until the algorithm converges, the value of the parameter W1 in the model can be determined, thereby completing the training of the model.

[0118] After completing the training of the image-text matching classification model, the role of the model is to input an image and text of a target object, and the classification model can give a result of whether the image and text of the target object match. What is needed in the embodiment of the present application is the image-text cross-features after the image features and text features of the target object are fused. Therefore, the output layer in the classification model can also be removed, and the image-text matching classification model can be used as a multimodal feature fusion model. That is, the multimodal feature fusion model can be obtained according to the feature fusion module in the image-text matching model. In this way, when the image and text content of a target object are also used as input, its output can be the image-text cross-features after the image features and text features are fused.

[0119] In this way, after the target object group is determined, each target object in the group can be input into the feature generation model, and the text features and image features of each target object can be extracted, and the image-text cross-features of each target object can be obtained. Afterwards, the image-text cross-features of each target object can be used to determine whether each target object has commonality in target attributes.

[0120] There are many forms of discriminant models for judging whether target objects have commonalities in target attributes based on the cross-image and text features. For example, in a simple way, the specific discriminant model can be to calculate the distance between the cross-image and text feature vectors of each target object (for example, the Euclidean distance between vectors, etc.), and then judge whether two target objects belong to the same / similar product objects based on the specific calculated distance and a preset threshold.

[0121] In order to further improve the accuracy of recognition, in a preferred embodiment of the present application, the discriminant model can also be generated by supervised learning training. For example, a discriminant model can be pre-built, which can take the cross-text features of multiple target objects as input and output the judgment result of whether the multiple target objects have commonalities in the target attributes (for example, whether the commodity objects are the same or similar, etc.).

[0122] Specifically, when the above-mentioned discriminant model is trained for supervised learning, multiple target object groups (for example, commodity object A and commodity object B form a commodity object pair, etc.) can be obtained in advance as training samples, and the image-text cross-features of each target object in the commodity object group can be obtained in advance (for example, obtained in advance using a previously trained feature generation model), and annotation information about whether each target object in the same group has commonality in target attributes is added. Then, the discriminant model is trained using such training samples, so that after the image-text cross-features of a group of target objects are input to the discriminant model, a discriminant result about whether the group of target objects has commonality in target attributes can be output.

[0123] In addition, since the specific discriminant model uses the cross-text and image features of the target object as input for discrimination, in fact, before the discrimination, it involves the process of extracting and fusing the text features and image features of each target object to generate the cross-text and image features. In specific implementation, the feature generation model and the above-mentioned discriminant model can be trained separately for supervised learning, or, in the preferred embodiment of the present application, in order to improve the training efficiency and further improve the accuracy of the final model recognition, the feature generation model and the discriminant model can be directly combined together (for example, a discriminant model is spliced ​​after the feature generation model, etc.), so as to generate a complete set of target models, which can directly use the text and image content of multiple target objects as input, and output the judgment result of whether multiple target objects have commonalities in target attributes.

[0124] In addition, when the target model is combined into a complete set, the feature generation model and the discriminant model can be trained together. For example, when training, Figure 4As shown, multiple groups of commodity objects (for example, commodity object 1 and commodity object 2 form a commodity object pair, etc.) can be directly obtained as training samples, and annotation information about whether each commodity object in the same group is the same or similar can be added. After that, the feature generation model can be used to extract text features and image features from commodity object 1 and commodity object 2 respectively, and then fuse them to generate image-text cross feature vectors, which are vector M and vector N respectively. After that, the vector M and vector N can be input into the discriminant function for judgment, and the output judgment result can be brought into the loss function, and the parameters in the model can be solved using algorithms such as gradient descent, etc.

[0125] Among them, in order to further improve the performance of the model, the feature generation model can be trained separately by supervised learning first, and the parameter learning results in the feature generation model can be obtained according to the training results; for example, the parameter W1 in the above example can be determined by training the feature generation model separately. After that, supervised learning is performed again on the whole set of target models composed of the feature generation model and the discriminant model, and the parameter values ​​determined when the feature generation model was trained separately before can be used as the initial values ​​of the parameters in the target model, and then the target model is trained again by supervised learning. In other words, when training the above whole set of target models, random parameter initial values ​​are no longer used, but the parameter values ​​determined in the process of training the feature generation model separately are used as the initial values, which can further improve the performance of the model and improve the final recognition accuracy.

[0126] In summary, through the embodiments of the present application, the target object's multimodal features (for example, including text information and image information, etc.) can be fused to obtain the target object's cross-image features, and then the target object can be more accurately identified based on the cross-image features whether there are commonalities in the target attributes. Among them, when fusing the multimodal features, considering that the image features and the text features belong to different domains, the image features and the text features are not directly spliced, or simple mathematical operations are performed, but a multimodal feature fusion model can be constructed, and the model can be trained in a supervised learning manner, so that the multimodal feature fusion model can be used to fuse the text features and the image features. In order to realize the supervised learning training of the multimodal feature fusion model, it can also be converted into the training of the image-text matching classification model, that is, the specific multimodal feature fusion model can be in the process of performing the task of judging whether the text features and the image features match, the text features and the image features are fused, and then the fusion processing result can be determined as the cross-image feature. In this way, a supervised multimodal feature fusion model can be generated to generate graphic and text cross-features that can better express the characteristics of the target object, and then more accurately identify whether the target objects have commonalities in target attributes (for example, whether the product objects belong to the same / similar models).

[0127] In addition, in actual applications, in some scenarios, the user can also be provided with recognition results on whether the target objects have commonalities in the target attributes, and the user can be provided with an operation option to provide feedback on the recognition results. For example, in the "find similar" scenario, after the user initiates a "find similar" request for a certain commodity object A, the recognition result can be returned through model calculation, for example, it can include commodity objects B, C, D, etc., and a "feedback" operation option can also be given for each commodity object. Assuming that the user finds that commodity object C is not the same as commodity object A, feedback information can be submitted, etc. After receiving user feedback information on the accuracy of the recognition result, the target model can also be updated and iterated using this user feedback information. In other words, the information fed back by the user can also become a labeled sample, which can participate in the iterative training of the model.

[0128] It should be noted here that, in the specific implementation, in the scenario where the commodity object is the target object, different target models can also be trained according to multiple different commodity object categories, so as to use the target models of specific categories to perform more accurate identification of the same / similar commodity objects. In this way, the text information and image information of each commodity object can be input into the target model corresponding to the target category for processing according to the common target category to which each commodity object in the target commodity object group belongs. For example, when it is specifically necessary to find the same or similar commodity object for a certain commodity object A, the category to which the commodity object A belongs can be first determined (the category information to which the specific commodity object belongs will be stored in the commodity object information library, so the category information of the commodity object can be obtained by querying the commodity object information library), and the commodity object is first screened according to the category information from the commodity library, and then, the target model corresponding to the category is used to identify whether the commodity object A and other commodity objects of the same category are the same / similar, and so on.

[0129] Embodiment 2

[0130] The second embodiment corresponds to the first embodiment, and provides a model processing method for the specific process of training the multimodal feature fusion model. Figure 5 , the method may include:

[0131] S501: constructing an image-text matching model, wherein the image-text matching model is used to identify whether an image and a text are used to describe a unified object;

[0132] S502: Acquire a training sample, wherein the training sample includes a plurality of sample entries, wherein the sample entries include image content and text content, and annotation information on whether the image content matches the text content;

[0133] S503: training the image-text matching model using the training samples;

[0134] S504: After the training is completed, the multimodal feature fusion model is generated according to the feature fusion module in the image-text matching model to extract image features and text features of the input target object, and the image text is fused with the text features to generate image-text cross features.

[0135] Embodiment 3

[0136] The third embodiment is also corresponding to the first embodiment. In the training process of the target model (composed of a feature generation model and a discriminant model), a model processing method is provided. Figure 6 , the method may include:

[0137] S601: training the feature generation model to determine parameter values ​​therein; the feature generation model is used to fuse the image features and text features of the target object to generate image-text cross features;

[0138] S602: Acquire a training sample, wherein the training sample includes a plurality of target object groups, each target object group includes at least two target objects, and annotation information on whether the at least two target objects have commonality in target attributes;

[0139] S603: Train the target model using the training samples, and use the parameter values ​​determined during the training of the feature generation model as initial values ​​of the parameters when training the target model.

[0140] Embodiment 4

[0141] This fourth embodiment provides a method for providing a recognition result for a specific application scenario. Figure 7 , the method may include:

[0142] S701: receiving text information and image information of an object to be identified submitted by a user, wherein the image information is generated by capturing an image of a physical object corresponding to the object to be identified;

[0143] S702: The object to be identified and a plurality of commodity objects in a commodity object library are respectively formed into a target object group; the commodity objects in the commodity object library are also associated with tags, and the tags are used to indicate whether the corresponding commodity object is authentic or how close it is to the authentic;

[0144] S703: Inputting the text information and image information of each target object in the target object group into a target model for processing, wherein the target model is used to fuse the text features and image features of each target object respectively, and determine the similarity between the target objects according to the obtained text-image cross feature; wherein the target model includes a multimodal feature fusion model, and the multimodal feature fusion model is used to fuse the text features and image features in the process of performing the task of judging whether the text features and the image features match, so as to obtain the text-image cross feature;

[0145] S704: providing recognition result information on whether the object to be identified is authentic or how close it is to the authentic product based on the label corresponding to the target commodity object whose similarity with the object to be identified meets the conditions.

[0146] For the undescribed parts in the above-mentioned embodiments 2 to 4, please refer to the description of embodiment 1 and other parts in this specification, which will not be repeated here.

[0147] It should be noted that the embodiments of the present application may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described herein within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

[0148] Corresponding to the first embodiment, the present application embodiment also provides an object recognition device, see Figure 8 , the device may include:

[0149] A target object group determination unit 801 is used to determine a target object group to be identified, wherein the target object group includes at least two target objects to be identified, and the target objects are associated with text information and image information;

[0150] The recognition processing unit 802 is used to input the text information and image information of each target object in the target object group into the target model for processing to obtain an object recognition result; wherein the target model is used to fuse the text features and image features of each target object respectively, and determine whether each target object has commonality in target attributes based on the obtained text-image cross features of each target object;

[0151] The target model includes a multimodal feature fusion model, and the multimodal feature fusion model is used to fuse the text features and image features in the process of performing a task of judging whether the text features and image features match, so as to obtain the image-text cross-features.

[0152] In a specific implementation, the target model includes a feature generation model and a discrimination model, and the feature generation model includes a feature extraction model and a multimodal feature fusion model;

[0153] The feature extraction model is used to: extract text features and image features from each target object respectively;

[0154] The multimodal feature fusion model is used to: fuse text features with image features and output image-text cross features of each target object;

[0155] The discriminant model is used to determine whether each target object has commonality in target attributes according to the image-text cross-features of each target object.

[0156] The target model is a complete set of models formed by combining a feature generation model and a discrimination model.

[0157] In a specific implementation, before training the target model, the feature generation model can also be trained, and the parameter learning results in the feature generation model can be obtained based on the training results; then, the parameter learning results are used as the initial parameter values ​​when training the target model.

[0158] In a specific implementation, the device may further include:

[0159] The result feedback unit is used to provide a recognition result on whether the target object has commonality in the target attribute, receive user feedback information on the accuracy of the recognition result, and iteratively train the target model according to the user feedback information.

[0160] Wherein, the target object includes a commodity object;

[0161] The identification processing unit is specifically used to determine whether each commodity object in the same commodity object group belongs to the same or similar commodity objects.

[0162] Wherein, at least two commodity objects included in the commodity object group correspond to the same category;

[0163] The target model includes a plurality of different target models corresponding to different categories;

[0164] The identification processing unit can be specifically used for:

[0165] According to the target category to which each commodity object in the target commodity object group belongs, the text information and image information of each commodity object are input into the target model corresponding to the target category for processing.

[0166] The target object group determination unit may be specifically used for:

[0167] When recommending commodity objects based on commodity objects that the user has browsed historically, the commodity objects that have been browsed historically and other data objects in the commodity object library are combined into a plurality of target commodity object groups.

[0168] Alternatively, the target object group determination unit may be specifically configured to:

[0169] The step of determining the target object group to be identified includes:

[0170] After receiving a request submitted by a user to search for a product object of the same item or similar item as a target product object, the target product object and other data objects in the product object library are combined into a plurality of target product object groups.

[0171] Alternatively, the target object group determination unit may be specifically configured to:

[0172] Receiving text information and image information of an object to be identified submitted by a user, wherein the image information is generated by capturing an image of a physical object corresponding to the object to be identified;

[0173] The object to be identified and a plurality of commodity objects in a commodity object library are respectively formed into a target object group; the commodity objects in the commodity object library are also associated with tags, and the tags are used to indicate whether the corresponding commodity objects are genuine or how close they are to the genuine ones;

[0174] The device may also include:

[0175] The authentic product identification result providing unit is used to provide identification result information on whether the object to be identified is authentic or how close it is to the authentic product based on the label corresponding to the target commodity object whose similarity with the object to be identified meets the conditions.

[0176] Corresponding to the second embodiment, the present application embodiment also provides a model processing device, see Fig. 9 , the device may include:

[0177] A model building unit 901 is used to build an image-text matching model, wherein the image-text matching model is used to identify whether an image and a text are used to describe a unified object;

[0178] A training sample acquisition unit 902 is used to acquire a training sample, wherein the training sample includes a plurality of sample entries, wherein the sample entries include image content and text content, and annotation information on whether the image content matches the text content;

[0179] A training unit 903, configured to train the image-text matching model using the training samples;

[0180] The multimodal feature fusion model generation unit 904 is used to generate the multimodal feature fusion model after the training is completed, according to the feature fusion module in the image-text matching model, so as to extract image features and text features of the input target object, and fuse the image text with the text features to generate image-text cross features.

[0181] Corresponding to the third embodiment, the present application embodiment further provides a model processing device, wherein the model is composed of a feature generation model and a discrimination model. Fig.10 , the device comprises:

[0182] The first training unit 1001 is used to train the feature generation model and determine the parameter values ​​therein; the feature generation model is used to fuse the image features and text features of the target object to generate image-text cross features;

[0183] The training sample acquisition unit 1002 is used to acquire training samples, wherein the training samples include multiple target object groups, each target object group includes at least two target objects, and annotation information about whether the at least two target objects have commonality in target attributes, wherein each commodity object is associated with text information and image information;

[0184] The second training unit 1003 is used to train the target model using the training samples, wherein the parameter values ​​determined during the training of the feature generation model are used as initial values ​​of the parameters when training the target model.

[0185] Corresponding to the fourth embodiment, the present application embodiment also provides a device for providing recognition results, see Fig.11 , the device may include:

[0186] The information receiving unit 1101 is used to receive text information and image information of an object to be identified submitted by a user, wherein the image information is generated by capturing an image of a physical object corresponding to the object to be identified;

[0187] The target object group generating unit 1102 is used to form target object groups with the to-be-identified object and a plurality of commodity objects in a commodity object library; the commodity objects in the commodity object library are also associated with tags, and the tags are used to indicate whether the corresponding commodity objects are genuine or how close they are to the genuine ones;

[0188] The similarity recognition unit 1103 is used to input the text information and image information of each target object in the target object group into the target model for processing, and the target model is used to fuse the text features and image features of each target object respectively, and determine the similarity between the target objects according to the obtained image-text cross feature; wherein the target model includes a multimodal feature fusion model, and the multimodal feature fusion model is used to fuse the text features and image features in the process of performing the task of judging whether the text features and the image features match, so as to obtain the image-text cross feature;

[0189] The recognition result providing unit 1104 is used to provide recognition result information on whether the object to be recognized is authentic or how close it is to the authentic product based on the label corresponding to the target commodity object whose similarity with the object to be recognized meets the conditions.

[0190] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0191] And an electronic device, comprising:

[0192] one or more processors; and

[0193] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0194] in, Fig.12 The architecture of the electronic device is shown as an example, which may include a processor 1210, a video display adapter 1211, a disk drive 1212, an input / output interface 1213, a network interface 1214, and a memory 1220. The processor 1210, the video display adapter 1211, the disk drive 1212, the input / output interface 1213, the network interface 1214, and the memory 1220 may be communicatively connected via a communication bus 1230.

[0195] Among them, the processor 1210 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solution provided in this application.

[0196] The memory 1220 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1220 can store an operating system 1221 for controlling the operation of the electronic device 1200, and a basic input and output system (BIOS) for controlling the low-level operation of the electronic device 1200. In addition, a web browser 1223, a data storage management system 1224, and an object recognition processing system 1225, etc. can also be stored. The above-mentioned object recognition processing system 1225 can be an application program that specifically implements the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 1220 and is called and executed by the processor 1210.

[0197] The input / output interface 1213 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0198] The network interface 1214 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0199] The bus 1230 comprises a pathway for transmitting information between the various components of the device (eg, the processor 1210 , the video display adapter 1211 , the disk drive 1212 , the input / output interface 1213 , the network interface 1214 , and the memory 1220 ).

[0200] It should be noted that, although the above device only shows a processor 1210, a video display adapter 1211, a disk drive 1212, an input / output interface 1213, a network interface 1214, a memory 1220, a bus 1230, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0201] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0202] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0203] The object recognition method and electronic device provided by the present application are introduced in detail above. The principle and implementation method of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. An object recognition method, characterized in that: include: Determine a target object group to be identified, wherein the target object group includes at least two target objects to be identified, and the target objects are associated with text information and image information; Inputting the text information and image information of each target object in the target object group into the target model for recognition processing to obtain an object recognition result; wherein the target model is used to fuse the text features and image features of each target object respectively, and judging whether each target object has commonality in target attributes based on the obtained text-image cross features of each target object; The target model includes a feature generation model and a discrimination model. The feature generation model includes a feature extraction model and a multimodal feature fusion model. In the process of performing the task of judging whether text features and image features match, the feature extraction model is used to extract text features and image features of each target object respectively. The multimodal feature fusion model is used to fuse the text features with the image features and output the text-image cross-features of each target object. The discrimination model is used to judge whether each target object has commonalities in target attributes based on the text-image cross-features of each target object.

2. The method according to claim 1, characterized in that: The target model is a set of models formed by combining a feature generation model and a discrimination model.

3. The method according to claim 2, characterized in that Before training the target model, the method further includes: Training the feature generation model, and obtaining parameter learning results in the feature generation model according to the training results; The parameter learning results are used as initial parameter values ​​when training the target model.

4. The method according to claim 1, characterized in that Also includes: Provide a recognition result on whether the target object has commonality in the target attribute, receive user feedback information on the accuracy of the recognition result, and iteratively train the target model according to the user feedback information.

5. The method according to any one of claims 1 to 4, characterized in that: The target object includes a commodity object; The determining whether the objects have commonality in the target attribute includes: Determine whether each product object in the same product object group is a product object of the same or similar type.

6. The method according to claim 5, characterized in that At least two commodity objects included in the commodity object group correspond to the same category; The target model includes a plurality of different target models corresponding to different categories; The step of inputting the text information and image information of each target object in the target object group into the target model for processing includes: According to the target category to which each commodity object in the target commodity object group belongs, the text information and image information of each commodity object are input into the target model corresponding to the target category for processing.

7. The method according to claim 5, characterized in that The step of determining the target object group to be identified includes: When recommending commodity objects based on commodity objects that the user has browsed historically, the commodity objects that have been browsed historically and other data objects in the commodity object library are combined into a plurality of target commodity object groups.

8. The method according to claim 5, characterized in that The step of determining the target object group to be identified includes: After receiving a request submitted by a user to search for a product object of the same item or similar item as a target product object, the target product object and other data objects in the product object library are combined into a plurality of target product object groups.

9. The method according to any one of claims 1 to 4, characterized in that: The step of determining the target object group to be identified includes: Receiving text information and image information of an object to be identified submitted by a user, wherein the image information is generated by capturing an image of a physical object corresponding to the object to be identified; The object to be identified and a plurality of commodity objects in a commodity object library are respectively formed into a target object group; the commodity objects in the commodity object library are also associated with tags, and the tags are used to indicate whether the corresponding commodity objects are genuine or how close they are to the genuine ones; The method further comprises: According to the label corresponding to the target commodity object whose similarity with the object to be identified meets the conditions, identification result information on whether the object to be identified is an authentic product or the degree of similarity to the authentic product is provided.

10. A model processing method, characterized in that: include: Constructing an image-text matching classification model, wherein the image-text matching classification model is used to identify whether the image and the text are used to describe the same object; The image-text matching classification model includes a feature fusion module and an output layer, wherein the output layer is used to output a recognition result of whether the image and text of the target object match; Acquire a training sample, wherein the training sample includes a plurality of sample entries, wherein the sample entries include image content and text content, and annotation information regarding whether the image content and the text content match; Training the image-text matching classification model through the training samples; After the training is completed, the output layer in the image-text matching classification model is removed so as to generate a multimodal feature fusion model according to the feature fusion module in the image-text matching classification model; wherein the multimodal feature fusion model and the feature extraction model constitute a feature generation model, and the feature generation model and the discrimination model constitute a target model; in the process of executing the task of judging whether the text features and the image features match through the target model, the feature extraction model is used to extract image features and text features of the input target object, and the multimodal feature fusion model is used to fuse the image text with the text features and output the image-text cross-features of the target object; the discrimination model is used to judge whether each target object has commonalities in the target attributes according to the image-text cross-features of each target object.

11. A method for providing recognition results, characterized in that: include: Receiving text information and image information of an object to be identified submitted by a user, wherein the image information is generated by capturing an image of a physical object corresponding to the object to be identified; The object to be identified and a plurality of commodity objects in a commodity object library are respectively formed into a target object group; the commodity objects in the commodity object library are also associated with tags, and the tags are used to indicate whether the corresponding commodity objects are genuine or how close they are to the genuine ones; The text information and image information of each target object in the target object group are input into the target model for processing, and the target model is used to fuse the text features and image features of each target object respectively, and determine the similarity between the target objects according to the obtained image-text cross-features; wherein the target model includes a feature generation model and a discrimination model, and the feature generation model includes a feature extraction model and a multimodal feature fusion model. In the process of performing the task of judging whether the text features and the image features match, the feature extraction model is used to extract text features and image features from each target object respectively, and the multimodal feature fusion model is used to fuse the text features and image features, and output the image-text cross-features of each target object; the discrimination model is used to judge whether each target object has commonality in target attributes according to the image-text cross-features of each target object; According to the label corresponding to the target commodity object whose similarity with the object to be identified meets the conditions, identification result information on whether the object to be identified is an authentic product or the degree of similarity to the authentic product is provided.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 11 are implemented.

13. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of claims 1 to 11.

Images