Information processing method and device and commodity query method and device

Abstract

The embodiment of the invention provides an information processing method and device and a commodity query method and device, and the information processing method comprises the steps: obtaining image-text query information, and determining an information attribute type corresponding to the image-text query information; determining image query information and text query information in the image-text query information, and constructing image-text fusion information according to the image query information and the text query information; carrying out object recall on the image query information, the text query information and the image-text fusion information to obtain an image recall object, a text recall object and an image-text recall object; and sorting the image recall object, the text recall object and the image-text recall object according to the information attribute type, and determining a target object corresponding to the image-text query information according to a sorting result.

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of information processing technology, and in particular, to information processing methods and devices, and commodity query methods and devices. Background Art

[0002] With the development of computer and Internet technology, query services are applied in more and more scenarios. For example, online shopping scenarios, question-and-answer scenarios, order query scenarios, information retrieval scenarios, etc., all involve the use of query services. As users' query needs continue to increase, various platforms that support query services will support two types of query methods: images and texts. Users can complete the query of relevant information on the platform by uploading images or editing detailed information, thereby meeting user query needs. In the prior art, although query services can be provided to users in the form of images or texts, the query results of a single image or a single text do not necessarily meet user needs, and this type of query method covers a smaller information surface and has a lower richness of query results. Therefore, an effective solution is urgently needed to solve the above problems. Summary of the invention

[0003] In view of this, the embodiments of this specification provide an information processing method. One or more embodiments of this specification also relate to an information processing device, a commodity query method, a commodity query device, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, there is provided an information processing method, including:

[0005] Acquire image and text query information, and determine the information attribute type corresponding to the image and text query information;

[0006] Determining image query information and text query information in the image-text query information, and constructing image-text fusion information according to the image query information and the text query information;

[0007] Recalling objects respectively for the image query information, the text query information and the image-text fusion information to obtain image recall objects, text recall objects and image-text recall objects;

[0008] The image recall objects, the text recall objects and the graphic and text recall objects are sorted according to the information attribute type, and the target object corresponding to the graphic and text query information is determined according to the sorting result.

[0009] According to a second aspect of the embodiments of this specification, a commodity query method is provided, including:

[0010] Acquire image and text query information, and determine the information attribute type corresponding to the image and text query information;

[0011] Determining image query information and text query information in the image-text query information, and constructing image-text fusion information according to the image query information and the text query information;

[0012] Recall products respectively according to the image query information, the text query information and the image-text fusion information to obtain image-recalled products, text-recalled products and image-text-recalled products;

[0013] The image recalled products, the text recalled products and the picture and text recalled products are sorted according to the information attribute type, and the target products corresponding to the picture and text query information are determined according to the sorting result.

[0014] According to a third aspect of the embodiments of this specification, another commodity query method is provided, including:

[0015] Receive the image and text query information submitted by the user through the product query interface, and determine the information attribute type corresponding to the image and text query information;

[0016] Determining image query information and text query information in the image-text query information, and constructing image-text fusion information according to the image query information and the text query information;

[0017] Recall products respectively according to the image query information, the text query information and the image-text fusion information to obtain image-recalled products, text-recalled products and image-text-recalled products;

[0018] Sorting the image recalled products, the text recalled products and the picture and text recalled products according to the information attribute type, and determining the target product corresponding to the picture and text query information according to the sorting result;

[0019] The product query interface is updated to a product display interface including the target product, and is displayed to the user.

[0020] According to a fourth aspect of the embodiments of this specification, there is provided an information processing device, including:

[0021] An acquisition module is configured to acquire image and text query information and determine an information attribute type corresponding to the image and text query information;

[0022] A determination module, configured to determine image query information and text query information in the image-text query information, and construct image-text fusion information according to the image query information and the text query information;

[0023] A recall module is configured to perform object recall for the image query information, the text query information and the image-text fusion information respectively, and obtain an image recall object, a text recall object and an image-text recall object;

[0024] The sorting module is configured to sort the image recall objects, the text recall objects and the graphic and text recall objects according to the information attribute type, and determine the target object corresponding to the graphic and text query information according to the sorting result.

[0025] According to a fifth aspect of the embodiments of this specification, a commodity query device is provided, including:

[0026] An information acquisition module is configured to acquire image and text query information and determine an information attribute type corresponding to the image and text query information;

[0027] An information determination module is configured to determine image query information and text query information in the image-text query information, and construct image-text fusion information according to the image query information and the text query information;

[0028] A product recall module is configured to recall products according to the image query information, the text query information and the image-text fusion information, respectively, to obtain image-recalled products, text-recalled products and image-text-recalled products;

[0029] The product sorting module is configured to sort the image recall products, the text recall products and the picture and text recall products according to the information attribute type, and determine the target product corresponding to the picture and text query information according to the sorting result.

[0030] According to a sixth aspect of the embodiments of this specification, another commodity query device is provided, including:

[0031] An information receiving module is configured to receive the image and text query information submitted by the user through the product query interface, and determine the information attribute type corresponding to the image and text query information;

[0032] An information determination module is configured to determine image query information and text query information in the image-text query information, and construct image-text fusion information according to the image query information and the text query information;

[0033] A product recall module is configured to recall products according to the image query information, the text query information and the image-text fusion information, respectively, to obtain image-recalled products, text-recalled products and image-text-recalled products;

[0034] a product sorting module, configured to sort the image recall products, the text recall products and the graphic and text recall products according to the information attribute type, and determine the target product corresponding to the graphic and text query information according to the sorting result;

[0035] The commodity display module is configured to update the commodity query interface to a commodity display interface including the target commodity, and display the interface to the user.

[0036] According to a seventh aspect of an embodiment of this specification, there is provided a computing device, including:

[0037] Memory and processor;

[0038] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the above-mentioned information processing method or commodity query method are implemented.

[0039] According to an eighth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned information processing method or product query method are implemented.

[0040] According to a ninth aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions, which implement the steps of the above-mentioned information processing method or product query method when executed by a processor.

[0041] The information processing method provided in this embodiment provides a mixed image and text query function to the user in order to accurately meet the user's query needs. After obtaining the image and text query information input by the user, the information attribute type corresponding to the image and text query information can be determined first, and the priority of the query result can be reflected through the information attribute type. After that, the image query information and text query information in the image and text query information can be determined. At this time, the image and text fusion information can be constructed based on the image query information and the text query information, and the image query and text query information can be fused, so that the fusion information can be combined to recall the object later. On this basis, object recall can be performed for image query information, text query information and image-text fusion information respectively, so as to obtain image recall objects, text recall objects and image-text recall objects according to the recall results; on this basis, considering that the recall objects correspond to different query information, the correlations between them and the image-text query information are different, so in order to accurately hit the query intent of the image-text query information, the image recall objects, text recall objects and image-text recall objects can be sorted according to the information attribute type, so as to determine the target objects corresponding to the image-text query information according to the sorting results; when processing object queries, the user's query needs can be hit more accurately, so as to feedback more accurate object query results to the user for user use. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a schematic diagram of an information processing method provided by an embodiment of this specification;

[0043] Figure 2 is a flow chart of an information processing method provided by an embodiment of this specification;

[0044] Figure 3 It is a schematic diagram of a product selection interface in an information processing method provided by an embodiment of this specification;

[0045] Figure 4 is a flow chart of a commodity query method provided by an embodiment of this specification;

[0046] Figure 5 is a flow chart of another commodity query method provided by an embodiment of this specification;

[0047] Figure 6 is a processing flow chart of an information processing method provided by an embodiment of this specification;

[0048] Figure 7 is a structural diagram of an information processing device provided by an embodiment of this specification;

[0049] Figure 8 It is a structural schematic diagram of a commodity query device provided by an embodiment of this specification;

[0050] Fig. 9 It is a structural schematic diagram of another commodity query device provided by an embodiment of this specification;

[0051] Fig.10 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0052] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0053] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0054] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0055] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0056] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters. A large model can also be called a foundation model / foundation model. The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (LLM), a multi-modal pre-training model, etc.

[0057] When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0058] First, the terms involved in one or more embodiments of this specification are explained.

[0059] Multimodal large model: refers to a model that combines information from multiple modalities such as text, images, videos, and audio for training. This model can fuse information from multiple modalities to form a unified representation space. This fusion method enables the model to understand the input data more comprehensively, improve the accuracy and robustness of the model, thereby obtaining a more comprehensive and accurate understanding, thereby improving the performance and application scope of the model.

[0060] In this specification, an information processing method is provided. One or more embodiments of this specification simultaneously relate to an information processing device, a commodity query method, a commodity query device, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0061] See also Figure 1As shown in the schematic diagram, the information processing method provided by this embodiment provides the user with a mixed image and text query function in order to accurately meet the user's query needs. After obtaining the image and text query information input by the user, the information attribute type corresponding to the image and text query information can be determined first, and the priority of the query result can be reflected through the information attribute type. After that, the image query information and text query information in the image and text query information can be determined. At this time, the image and text fusion information can be constructed based on the image query information and the text query information, and the image query and text query information can be fused, so that the fusion information can be combined to recall the object later. On this basis, object recall can be performed for image query information, text query information and image-text fusion information respectively, so as to obtain image recall objects, text recall objects and image-text recall objects according to the recall results; on this basis, considering that the recall objects correspond to different query information, the correlations between them and the image-text query information are different, so in order to accurately hit the query intent of the image-text query information, the image recall objects, text recall objects and image-text recall objects can be sorted according to the information attribute type, so as to determine the target objects corresponding to the image-text query information according to the sorting results; when processing object queries, the user's query needs can be hit more accurately, so as to feedback more accurate object query results to the user for user use.

[0062] See also Figure 2 , Figure 2 A flowchart of an information processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0063] Step S202: Acquire image and text query information, and determine the information attribute type corresponding to the image and text query information.

[0064] The information processing method provided in this embodiment can be applied to any scenario with information query services, such as information query scenarios (searching platform information), online shopping scenarios (finding products), question-and-answer scenarios (knowledge questions and answers), etc. When users query related results, they can combine the image information and / or text information input by the user to complete the result query, and then feedback the results that hit their query needs to the user. This embodiment takes the information processing method of querying products in an online shopping scenario as an example to illustrate. The descriptions of other scenarios can refer to the same or corresponding descriptions in this embodiment, and this embodiment will not be repeated in detail here.

[0065] Specifically, the image-text query information specifically refers to the image-text query constructed by combining the image query information and text query information input by the user, which is used for subsequent result query, such as querying products, articles, orders, etc. Among them, the image query information specifically refers to the image input by the user based on his own needs. For example, in the product query scenario, the image can be the image of the product to be queried input by the user; for another example, in the article query scenario, the image can be a screenshot of the local content of the article input by the user. Correspondingly, the text query information specifically refers to the query intention description information input by the user query information, which can be understood as the query demand description text. For example, in the product query scenario, the text can be the text described by the user for the input image, or it can be the text described by the user independently of the image. Correspondingly, the information attribute type specifically refers to the type of the corresponding attribute of the image-text query information, which is used to reflect the attribute relationship between the image query information and the text query information in the image-text query information, such as the image-text consistent attribute type, the image-text inconsistent attribute type, the image-text attribute conflict type, the image-text attribute non-conflict type, etc.

[0066] Among them, the image-text consistent attribute type is used to represent that both the image query information and the text query information represent the same event; for example, the user inputs the image of a red skirt, and the description text is "Please query red skirt products"; the image-text inconsistent attribute type represents that the image query information and the text query information do not represent the same event; for example, the user inputs the image of a red skirt, and the description text is "Please query laptop products"; the image-text attribute conflict type refers to that the image and text bodies are consistent but the attributes are different, for example, the user inputs the image of a red skirt, and the description text is "Please query blue skirt products of the same style"; the image-text attribute non-conflict type refers to that the image and text bodies are consistent and the attributes are the same, for example, the user inputs the image of a red skirt, and the description text is "Please query red skirt products of the same style".

[0067] In actual applications, most platforms that provide online shopping functions support image search and text search for products. In order to meet user needs, some platforms also support filtering products based on the text information entered by the user on the basis of image search results, and then provide users with feedback on the products they need to search for. However, some user query requirements cannot be fully expressed by selecting text or images alone, especially in some scenarios with high query complexity. User needs are diverse and more complex. Failure to accurately hit user query requirements will greatly affect the user query experience. Therefore, an effective solution and solution to the above problems are urgently needed.

[0068] Based on this, in order to accurately meet the user's query needs, the user will be provided with a mixed image and text query function, so that after obtaining the image and text query information input by the user, the information attribute type corresponding to the image and text query information can be determined first, and the priority of the query result can be reflected through the information attribute type. After that, the image query information and text query information in the image and text query information can be determined. At this time, the image and text fusion information can be constructed according to the image query information and the text query information, and the image query and text query information can be fused, so that the fusion information can be combined to perform object recall in the future. On this basis, the object recall can be performed for the image query information, the text query information and the image and text fusion information respectively, and the image recall object, the text recall object and the image and text recall object can be obtained according to the recall results; on this basis, considering that the recall object corresponds to different query information, the correlation between it and the image and text query information is different, so in order to accurately hit the query intention of the image and text query information, the image recall object, the text recall object and the image and text recall object can be sorted according to the information attribute type, so as to determine the target object corresponding to the image and text query information according to the sorting result.

[0069] Furthermore, when obtaining the image-text query information, considering that the image query information input by the user may contain one or more object entities, in order to ensure that the image-text query information used subsequently can accurately complete the object query, the image query information can be identified to complete the extraction of related objects. In this embodiment, the specific implementation method is as follows:

[0070] The image query information and text query information submitted by the user are received, and object recognition is performed on the image query information; the associated object is determined according to the object recognition result, and the image and text query information is constructed based on the associated object and the text query information.

[0071] Specifically, object recognition refers to the operation of identifying the object subject contained in the image query information. In practical applications, the object recognition operation can be implemented through an image recognition model. After the image is input into the image recognition model, the image recognition model can detect the object subject contained in the image, and then feedback the region coordinates corresponding to the object subject based on the detection result. The image can be cropped based on the region coordinates to obtain the object subject. Correspondingly, the associated object specifically refers to the object that the user needs to query in the image after object recognition is performed on the image query information.

[0072] Based on this, in order to ensure that the image and text query information highly matches the user's query needs, so that the subsequent combination of image query information and text query information can quickly and accurately query objects for use, after receiving the image query information and text query information submitted by the user, object recognition can be performed on the image query information; so as to determine the object subject that the user needs to query, that is, the associated object, according to the object recognition result; on this basis, the image and text query information can be constructed based on the associated object and text query information for subsequent use.

[0073] In practical applications, when the image query information contains only a single object subject, the image-text query information can be directly constructed based on the image and text, thereby improving the efficiency of object query.

[0074] In summary, by constructing image-text query information through object recognition on image query information, it can be ensured that the fusion of image and text better matches the user's query needs, thereby improving subsequent object query results.

[0075] Furthermore, when there are multiple candidate associated objects in the image query information, in order to avoid errors, the user can also interact to select an associated object for use. In this embodiment, the specific implementation is as follows:

[0076] When it is determined according to the recognition result that the image query information contains multiple candidate associated objects, selection items corresponding to the multiple candidate associated objects are fed back to the user; a selection request submitted by the user for the selection item is received, and an associated object is determined from the multiple candidate associated objects based on the selection request.

[0077] Specifically, the candidate associated objects specifically refer to multiple object entities obtained after object recognition of the image query information. Correspondingly, the selection items specifically refer to items fed back to the user for the user to determine the required query objects among multiple candidate associated objects. Correspondingly, the selection request is the selection request submitted by the user for the associated object.

[0078] Based on this, taking into account that the environment in which users take images in actual applications is relatively complex and may contain one or more object subjects, and different object subjects will affect the subsequent query results, therefore, when it is determined based on the recognition results that the image query information contains multiple candidate associated objects, the selection items corresponding to the multiple candidate associated objects can be fed back to the user at this time; for the user to make a selection, when receiving a selection request submitted by the user for the selection item, the associated object can be determined from the multiple candidate associated objects based on the selection request, so as to facilitate the subsequent construction of query information.

[0079] In summary, by using an interactive method to determine the associated objects that match the user's query requirements, the accuracy of constructing graphic and text query information can be further improved.

[0080] After obtaining the associated object and text query information, the two can be rewritten through the multimodal information processing model, thereby ensuring that the obtained image and text query information better matches the user's object query intention and achieves full integration of images and text. In this embodiment, the specific implementation method is as follows:

[0081] The associated object and the text query information are input into a multimodal information processing model; object attribute information corresponding to the associated object is extracted through the multimodal information processing model, and graphic and text query information is constructed based on the object attribute information and the text query information.

[0082] Specifically, the multimodal information processing model refers to a large multimodal model that can fuse text, image, audio and other information to ensure that the fused result does not lose image attributes and text attributes, so as to meet subsequent use. Correspondingly, object attribute information refers to the attribute description information obtained after extracting attribute information from associated objects in the form of images. By fusing it with text query information, image and text query information that reflects image attributes and text attributes can be obtained.

[0083] Based on this, when fusing the associated objects and text query information, considering that the associated objects belong to the image type and the text query information is input into the text type, in order to realize cross-module information fusion, the associated objects and the text query information can be input into the multimodal information processing model; the object attribute information corresponding to the associated objects is extracted through the multimodal information processing model, and on this basis, the image and text query information can be constructed based on the object attribute information and the text query information, so as to realize information fusion to obtain the image and text query information matching the user, so as to facilitate subsequent use.

[0084] For example, when a user searches for desired shopping products through a shopping platform, the platform provides the user with an image input interface and a text input interface. Figure 3The image shown in (a) contains two products, gel pens and notebooks. At this time, the image can be recognized and the recognized gel pens and notebooks can be fed back to the user to ask the user what product he needs to search for. It is determined that the user needs to search for gel pens, and the text information entered by the user is "I want a red gel pen of the same style"; on this basis, the multimodal large model can be used to understand the user's graphic and text needs, and the gel pen product body and the attribute information corresponding to the product body (press type, black gel pen, brand A, etc.) can be extracted from the image. After that, the attribute information can be fused with the text information to obtain the graphic and text semantic query, so that the graphic and text semantic query can be combined to complete the feedback of the user's product search needs.

[0085] In summary, by adopting a multimodal information processing model to fuse text and image information, we can obtain query information containing image and text semantics, so as to more accurately hit the target object in the future.

[0086] After the construction of the above-mentioned graphic and text query information is completed, considering that the information attribute type corresponding to the graphic and text query information is the basis for adjusting the sorting order of the recalled objects in the future, different information attribute types will determine different sorting methods, so the information type determination can be completed by extracting object information for comparison. In this embodiment, the specific implementation method is as follows:

[0087] Determine the first object information corresponding to the image query information according to the object recognition result, and perform entity extraction on the text query information to determine the second object information corresponding to the text query information; detect whether there is an attribute conflict between the image query information and the text query information according to the first object information and the second object information; if so, determine that the information attribute type corresponding to the image and text query information is an image-text conflict attribute type; if not, determine that the information attribute type corresponding to the image and text query information is an image-text non-conflict attribute type.

[0088] Specifically, the first object information specifically refers to the object description information determined after object recognition is performed on the image query information, which is used to describe the attributes of the object, and the second object information specifically refers to the corresponding object description information in the associated text query information determined after entity extraction is performed on the text query information. Correspondingly, the image-text conflict attribute type specifically refers to the type in which the image query information and the text query information have attribute conflicts, and conversely, the image-text non-conflict attribute type specifically refers to the type in which the image query information and the text query information do not have attribute conflicts.

[0089] Based on this, after obtaining the image and text query information, object recognition can be performed on the image query information in the image and text query information to determine the first object information corresponding to the image query information based on the object recognition result, and entity extraction can be performed on the text query information to determine the second object information corresponding to the text query information.

[0090] On this basis, the first object information and the second object information can be compared. If the first object information and the second object information are consistent, it means that the image query information and the text query information submitted by the user are consistent descriptions. For example, if a user uploads an image of a red skirt and the description text is "Please search for skirts of the same style and color", then it is determined that there is no conflict between the image and the text. Therefore, it can be determined that the information attribute type corresponding to the image and text query information is an image-text non-conflict attribute type. When the recalled objects are sorted later, the weights of the image and text dimensions can be increased, and the weights of the image and text dimensions can be reduced, thereby ensuring the accuracy of the recall results.

[0091] If the first object information and the second object information are inconsistent, it means that the image query information and text query information submitted by the user are inconsistent descriptions. For example, if a user uploads an image of a red skirt and the description text is "Please search for a blue skirt of the same style", it is determined that there is a conflict between the image and the text. Therefore, it can be determined that the information attribute type corresponding to the image and text query information is the image and text conflict attribute type. When the recalled objects are sorted later, the weight of the image and text dimension can be increased, and the weight of the image and text dimension can be reduced, thereby ensuring the accuracy of the recall results.

[0092] In summary, by determining whether the first object information and the second object information are consistent to determine whether the information attribute type is conflicting or not, it is possible to improve sorting accuracy when sorting objects subsequently.

[0093] Step S204: determining the image query information and the text query information in the image and text query information, and constructing image and text fusion information according to the image query information and the text query information.

[0094] Specifically, after obtaining the above-mentioned image-text query information, in order to realize object recall from multiple dimensions and thus improve the recall coverage, the image query information and text query information in the image-text query information can be determined, and the image-text fusion information can be constructed based on the image query information and the text query information, so as to realize subsequent object recall from the image dimension, the text dimension and the image-text dimension to improve the richness of the recall, and on this basis, the target object can be screened to effectively ensure the query hit accuracy. Among them, the image-text fusion information specifically refers to the vector expression obtained by fusing the image query information and the text query information.

[0095] Furthermore, when constructing the image-text fusion information, considering that the object recall operation will be performed based on the information later, and in order to make the recall result more focused on the image-text fusion attribute, the image-text fusion information can be determined by feature fusion. In this embodiment, the specific implementation method is as follows:

[0096] The image query information and the text query information are input into a graph-text representation model; the image query features corresponding to the image query information and the text query features corresponding to the text query information are extracted through the graph-text representation model; feature fusion is performed on the image query features and the text query features to obtain graph-text fusion features, and the graph-text fusion features are used as the graph-text fusion information.

[0097] Specifically, the image-text representation model specifically refers to a model that can fuse image features and text features, which can be implemented using the image-text representation model (BGE-visualized-base). Other image-text fusion models can also be selected in actual applications, and this embodiment does not make any restrictions here. Correspondingly, the image query feature is the feature expression corresponding to the image query information; correspondingly, the text query feature is the feature expression corresponding to the text query information. Correspondingly, the image-text fusion feature is the vector expression obtained by fusing the image query feature and the text query feature.

[0098] Based on this, considering that a single search method cannot accurately feedback the query object required by the user, for example, a user uploads an image of a red skirt and describes it as asking for a blue skirt of the same style. Usually, the product pool recalled by image search is mostly red skirts, and the product pool recalled by text search is mostly blue skirts, and most of them are different from the original image style. This is because although the image and text query information integrates part of the image and text, it ignores a large amount of image semantics.

[0099] Therefore, in order to achieve cross-modal information fusion, the image query information and the text query information can be input into the image-text representation model; the image query features corresponding to the image query information and the text query features corresponding to the text query information are extracted through the image-text representation model; on this basis, the image query features and the text query features are fused to obtain the image-text fusion features, and the image-text fusion features are used as the image-text fusion information to achieve full fusion of image semantics and text semantics, thereby ensuring that the subsequent recall of objects is more accurate and can hit the objects ignored by the image recall and text recall.

[0100] Continuing with the above example, after determining the image-text semantic query, in order to achieve multimodal recall in addition to image recall and text recall, the gel pen image and text information "I want the same style of red gel pen" can be input into the image-text representation model to extract the image feature V1 of the gel pen image and the text feature V2 of the text "I want the same style of red gel pen". Then, by fusing the image feature V1 and the text feature V2, the image-text feature V can be obtained, so that the product recall operation can be completed by combining the image, text and image-text features.

[0101] In summary, by adopting the image-text representation model to fuse image query information and text query information, information fusion at the semantic level can be achieved, which is more convenient for subsequent use.

[0102] Step S206 , performing object recall for the image query information, the text query information and the image-text fusion information respectively, and obtaining image recall objects, text recall objects and image-text recall objects.

[0103] Specifically, after obtaining the image query information, text query information, and image-text fusion information, in order to ensure the richness of object recall, the image query information, text query information, and image-text fusion information can be combined to perform object recall respectively. In other words, the image recall object can be recalled based on the image query information alone, the text recall object can be recalled based on the text query information, and the image-text recall object can be recalled based on the image-text fusion information, so that the objects recalled by each information can be sorted later, and then the target object corresponding to the image-text query information can be screened.

[0104] Among them, the image recall object specifically refers to the object recalled according to the image query information, the text recall object specifically refers to the object recalled according to the text query information, the picture and text recall object specifically refers to the object recalled according to the picture and text fusion information, and the picture and text recall operation can recall objects with conflicting picture and text attributes, thereby improving the richness of the recalled objects. Among them, the object refers to the result required for the query in the current scenario. For example, in an online shopping scenario, the object can be a recalled product; in a text search scenario, the object can be a recalled article; in an image search scenario, the object can be a recalled image, etc. In actual applications, the type of object is set according to the actual application scenario, and this embodiment does not make any limitation here.

[0105] In specific implementation, when recalling objects according to images and texts, a candidate object set can be preset, and recall units corresponding to different modalities can be set at the same time, so that the recall unit can recall objects in the candidate object set according to the information of the corresponding modality, thereby obtaining image recall objects and text recall objects.

[0106] Furthermore, after obtaining the image query information, text query information, and image-text fusion information, considering that the image query information and text query information are both information corresponding to a single modality, the recall of objects is more convenient, and the image-text fusion information combines the two modalities of image and text. In order to improve the recall accuracy, the object recall can be completed by calculating the similarity of features. In this embodiment, the specific implementation method is as follows:

[0107] Determine the candidate object features corresponding to the candidate objects in the candidate object set, and input the candidate object features and the image-text fusion features into a multimodal object recall model; calculate the feature similarity between the image-text fusion features and the candidate object features through the multimodal object recall model; screen the target candidate objects in the candidate object set according to the feature similarity, and use the target candidate objects as image-text recall objects.

[0108] Specifically, the candidate object set specifically refers to a set of candidate objects to be recalled, such as a product set, a text set, an image set, etc. Correspondingly, the candidate object feature specifically refers to the vector expression corresponding to each candidate object in the candidate object set. Correspondingly, the multimodal object recall model specifically refers to a multimodal large model with object recall capability, which can be used to input the image-text fusion information and filter the recall objects that match the image-text fusion information in the candidate object set according to the image-text fusion information. Correspondingly, the feature similarity is the similarity between the image-text fusion information and the candidate object feature corresponding to each candidate object.

[0109] Based on this, when recalling objects based on image-text fusion information, in order to improve recall efficiency and accuracy, a multimodal object recall model can be combined to achieve it. Specifically, the candidate object features corresponding to the candidate objects in the candidate object set can be determined first, and the candidate object features and the image-text fusion features can be input into the multimodal object recall model; at this time, the feature similarity between the image-text fusion features and the candidate object features corresponding to each candidate object can be calculated through the multimodal object recall model; after obtaining the feature similarity, the candidate objects can be sorted from large to small according to the feature similarity, and then the topN image-text recall objects can be selected, so that the target objects can be screened in combination with the image recall objects and the text recall objects in the subsequent step.

[0110] Continuing with the above example, after obtaining the gel pen image, the text information "I want a red gel pen of the same style" and the graphic feature V, a three-way recall operation can be performed based on the above three types of information. According to the recall operation, the products 1, 2, 3 and 4 corresponding to the gel pen image will be obtained, and the products 3, 5, 6 and 7 corresponding to the text information "I want a red gel pen of the same style" will be obtained. In addition, the product recall operation will be performed based on the graphic feature V. At this time, the feature similarity between the graphic feature V and each product can be calculated, and the top 4 can be selected as the recalled products corresponding to the graphic feature V according to the feature similarity calculation results, and the graphic feature V can be determined to correspond to products 3, 8, 9 and 10. On this basis, the products that meet the user's query can be selected from products 1 to 10 in the future and recommended to the user for purchase.

[0111] In summary, by combining the model to calculate feature similarity, objects can be recalled for image-text fusion information. On the basis of image recall results and text recall results, objects that are easily overlooked by the two and meet user query requirements can be recalled, thereby ensuring the accuracy of object recall.

[0112] Step S208 , sorting the image recall objects, the text recall objects and the graphic and text recall objects according to the information attribute type, and determining the target object corresponding to the graphic and text query information according to the sorting result.

[0113] Specifically, after obtaining the image recall objects of the image query information recall, the text recall objects of the text query information recall, and the image and text recall objects of the image and text fusion information recall, considering that there are different situations of the objects of different modal information recall, and in order to be able to screen out the objects that hit the user's query needs, the image recall objects, text recall objects, and image and text recall objects can be sorted according to the information attribute type, so as to characterize the priority of the image and text query information of different objects according to the sorting results, so as to determine the target object corresponding to the image and text query information according to the sorting results, so as to feedback the target object that is most likely to be queried to the user. Among them, the target object is the result queried according to the image and text query information, such as text, commodity, image, etc.

[0114] In actual applications, when sorting image recall objects, text recall objects, and image and text recall objects, it is necessary to complete it according to the information attribute type. If the information attribute type is the same attribute type as the image and text, the image recall objects are sorted first. If the attribute type is inconsistent, the image and text recall objects can be sorted first. This ensures that the sorting results match the user's object query requirements.

[0115] Furthermore, when sorting the image recall objects, text recall objects, and graphic recall objects, considering that different recall operations recall different objects and the relevance of hitting the user's query requirements is also different, the score weights can be adjusted according to the information attribute type, thereby ensuring that the sorting results better match the user's requirements. In this embodiment, the specific implementation method is as follows:

[0116] According to a preset sorting strategy, the image recall score corresponding to the image recall object, the text recall score corresponding to the text recall object, and the picture-text recall score corresponding to the picture-text recall object are determined; based on the information attribute type, the image recall score, the text recall score, and the picture-text recall score are updated to obtain a target image recall score, a target text recall score, and a target picture-text recall score; based on the target image recall score, the target text recall score, and the target picture-text recall score, the image recall object, the text recall object, and the picture-text recall object are sorted.

[0117] Specifically, the image recall score refers to the score of the relevance between the image query information and the image and text query information after scoring the object of the image query information recall, the text recall score refers to the score of the relevance between the text query information and the image and text query information after scoring the object of the image and text fusion information recall, and the image and text recall score refers to the score of the relevance between the image and text query information after scoring the object of the image and text fusion information recall. Correspondingly, the target image recall score, target text recall score, and target image and text recall score refer to the scores obtained by weighted adjustment of the recall scores based on the information attribute type, and the scores can reflect the matching degree between each recalled object and the image and text query information.

[0118] Based on this, after obtaining the objects of each information recall, taking into account that different information attribute types will reflect different query intentions of users, and in order to be able to filter the target objects in combination with the intention, the image recall score corresponding to the image recall object, the text recall score corresponding to the text recall object, and the picture-text recall score corresponding to the picture-text recall object can be determined according to the preset sorting strategy; on this basis, the image recall score, the text recall score, and the picture-text recall score can be updated based on the information attribute type to achieve the target image recall score, the target text recall score, and the target picture-text recall score that match the user's query requirements in a weighted manner; on this basis, the image recall objects, the text recall objects, and the picture-text recall objects can be sorted based on the target image recall score, the target text recall score, and the target picture-text recall score, so that the objects that match the user's query requirements can be sorted first, thereby facilitating the subsequent screening of target objects.

[0119] It should be noted that when calculating the image recall score corresponding to the image recall object, the image similarity, text similarity and picture-text similarity corresponding to the image recall object are actually calculated, and then the image similarity, text similarity and picture-text similarity corresponding to the image recall object are weighted summed, and then the image recall score corresponding to the image recall object can be obtained; similarly, the text recall score corresponding to the text recall object is also calculated. The image similarity, text similarity and picture-text similarity corresponding to the text recall object are then weighted summed, and then the text recall score corresponding to the text recall object can be obtained. The picture-text recall score corresponding to the picture-text recall object is also calculated. The image similarity, text similarity and picture-text similarity corresponding to the picture-text recall object are then weighted summed, and then the picture-text recall score corresponding to the picture-text recall object can be obtained. Based on this, the recall scores corresponding to each object are calculated, and the scores corresponding to the objects can be aligned, so that when the scores are subsequently sorted, the sorting results are more reasonable and accurate.

[0120] Furthermore, after the score is calculated, considering that different information attribute types will affect the final query results, the score composed of the three dimensions of similarity that has been calculated is weighted by adjusting the weights in combination with the information attribute type. This ensures that the final target score is more in line with the current query requirements, so that the target object can be accurately screened out and fed back to the user in the future.

[0121] In summary, by sorting the recalled objects in a weighted adjustment score manner, it can be ensured that the sorting results better match the user's query requirements, thereby improving the accuracy of target object screening.

[0122] Furthermore, in order to ensure that the target object finally fed back to the user highly meets the user's object query requirements, the recalled objects can also be filtered in combination with the object category information. In this embodiment, the specific implementation method is as follows:

[0123] The graphic and text query information is input into an object category prediction model for processing to obtain object category information corresponding to the graphic and text query information; wherein, determining the target object corresponding to the graphic and text query information according to the sorting result includes: determining an object sequence according to the sorting result; filtering the object sequence according to the object category information, and determining the target object corresponding to the graphic and text query information according to the filtering result.

[0124] Specifically, the object category prediction information refers to the information predicting the category to which the object to be queried belongs, and based on the object category information, objects that better match the user's query requirements can be screened out from the sorting results. Correspondingly, the object sequence is the sequence after the objects are sorted according to the scores.

[0125] Based on this, in order to improve the accuracy of determining the target object, the graphic and text query information can also be input into the object category prediction model for processing, so as to determine the object category information corresponding to the graphic and text query information according to the processing result. Thereafter, when determining the target object, the object sequence can be first determined according to the sorting result; then the object sequence can be filtered according to the object category information, so as to determine the target object corresponding to the graphic and text query information according to the filtering result.

[0126] Continuing with the above example, after determining products 1 to 10, it is determined that there is an inconsistency between the image and text input by the user. Therefore, the weight of the products recalled for the image and text feature V can be increased, and based on this, the similarity between each product and the user input information is calculated. According to the calculation results, the similarity ranking is determined to be product 3>product 2>product 1>product 4>product 5>product 6>product 7>product 8>product 9>product 10; at the same time, it is determined that the product that the user needs to search for is a gel pen. At this time, products 1-10 can be filtered again to determine that products 3, product 2, product 1, product 4, product 5, product 6, product 7, and product 8 meet the user's query needs. At this time, the following can be displayed to the user. Figure 3 The product display interface shown in (b) is for users to purchase red gel pens.

[0127] The information processing method provided in this embodiment provides a mixed image and text query function to the user in order to accurately meet the user's query needs. After obtaining the image and text query information input by the user, the information attribute type corresponding to the image and text query information can be determined first, and the priority of the query result can be reflected through the information attribute type. After that, the image query information and text query information in the image and text query information can be determined. At this time, the image and text fusion information can be constructed based on the image query information and the text query information, and the image query and text query information can be fused, so that the fusion information can be combined to recall the object later. On this basis, object recall can be performed for image query information, text query information and image-text fusion information respectively, so as to obtain image recall objects, text recall objects and image-text recall objects according to the recall results; on this basis, considering that the recall objects correspond to different query information, the correlations between them and the image-text query information are different, so in order to accurately hit the query intent of the image-text query information, the image recall objects, text recall objects and image-text recall objects can be sorted according to the information attribute type, so as to determine the target objects corresponding to the image-text query information according to the sorting results; when processing object queries, the user's query needs can be hit more accurately, so as to feedback more accurate object query results to the user for user use.

[0128] See also Figure 4 , Figure 4 A flowchart of a commodity query method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0129] Step S402, obtaining image and text query information, and determining the information attribute type corresponding to the image and text query information;

[0130] Step S404, determining image query information and text query information in the image-text query information, and constructing image-text fusion information according to the image query information and the text query information;

[0131] Step S406, recalling products respectively according to the image query information, the text query information and the image-text fusion information, and obtaining image-recalled products, text-recalled products and image-text-recalled products;

[0132] Step S408 , sorting the image recalled products, the text recalled products and the picture and text recalled products according to the information attribute type, and determining the target product corresponding to the picture and text query information according to the sorting result.

[0133] For the contents not described in detail in the commodity query method provided in this embodiment, please refer to the same or corresponding description in the above embodiments, and this embodiment will not be described in detail here.

[0134] See also Figure 5 , Figure 5 A flowchart of another commodity query method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0135] Step S502, receiving the image and text query information submitted by the user through the product query interface, and determining the information attribute type corresponding to the image and text query information;

[0136] Step S504, determining image query information and text query information in the image-text query information, and constructing image-text fusion information according to the image query information and the text query information;

[0137] Step S506, recalling products respectively according to the image query information, the text query information and the image-text fusion information, and obtaining image-recalled products, text-recalled products and image-text-recalled products;

[0138] Step S508, sorting the image recalled products, the text recalled products and the picture and text recalled products according to the information attribute type, and determining the target product corresponding to the picture and text query information according to the sorting result;

[0139] Step S510: updating the product query interface to a product display interface including the target product, and displaying the interface to the user.

[0140] Specifically, in order to support a more flexible commodity purchasing experience for users, after receiving the image and text query information submitted by the user through the commodity query interface, the information attribute type corresponding to the image and text query information can be determined, and the priority of the query result can be reflected through the information attribute type. Thereafter, the image query information and text query information in the image and text query information can be determined, and at this time, the image and text fusion information can be constructed based on the image query information and the text query information, so as to achieve the fusion of the image query and the text query information, so as to realize the subsequent recall of commodities in combination with the fusion information. On this basis, products can be recalled respectively for image query information, text query information and image-text fusion information, so as to obtain image recalled products, text recalled products and image-text recalled products according to the recall results; on this basis, considering that the recalled products correspond to different query information, the correlation between them and the image-text query information is different, so in order to accurately hit the query intent of the image-text query information, the image recalled products, text recalled products and image-text recalled products can be sorted according to the information attribute type, so as to determine the target products corresponding to the image-text query information according to the sorting results; finally, the product query interface can be updated to a product display interface containing the target products, and displayed to users to facilitate user product purchases.

[0141] For the contents not described in detail in the other commodity query method provided in this embodiment, please refer to the same or corresponding description in the above embodiments, and this embodiment will not be described in detail here.

[0142] The following combination Figure 6 , taking the application of the information processing method provided in this specification in the document search scenario as an example, the information processing method is further described. Figure 6 A processing flow chart of an information processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0143] Step S602: receiving image query information and text query information submitted by a user, and performing object recognition on the image query information.

[0144] Step S604: determine the associated objects according to the object recognition result, and input the associated objects and text query information into the multimodal information processing model.

[0145] Step S606: extracting object attribute information corresponding to the associated object through the multimodal information processing model.

[0146] Step S608: constructing graphic and text query information based on the object attribute information and the text query information, and determining the information attribute type corresponding to the graphic and text query information.

[0147] Step S610: determining image query information and text query information in the image-text query information, and inputting the image query information and the text query information into the image-text representation model.

[0148] Step S612: extracting image query features corresponding to the image query information and text query features corresponding to the text query information through the image-text representation model.

[0149] Step S614, performing feature fusion on the image query feature and the text query feature to obtain an image-text fusion feature, and using the image-text fusion feature as image-text fusion information.

[0150] Step S616, performing object recall for the image query information, the text query information and the image-text fusion information respectively, and obtaining image recall objects, text recall objects and image-text recall objects.

[0151] Among them, object recall is performed on the image-text fusion information to obtain the image-text recall object, including: determining the candidate object features corresponding to the candidate objects in the candidate object set, and inputting the candidate object features and the image-text fusion features into the multimodal object recall model; calculating the feature similarity between the image-text fusion features and the candidate object features through the multimodal object recall model; screening the target candidate objects in the candidate object set according to the feature similarity, and using the target candidate objects as the image-text recall object.

[0152] Step S618, determining the image recall score corresponding to the image recall object, the text recall score corresponding to the text recall object, and the picture and text recall score corresponding to the picture and text recall object according to the preset sorting strategy.

[0153] Step S620, updating the image recall score, text recall score and graphic-text recall score based on the information attribute type to obtain a target image recall score, a target text recall score and a target graphic-text recall score.

[0154] Step S622, sorting the image recall objects, text recall objects and picture-text recall objects based on the target image recall score, the target text recall score and the target picture-text recall score, and determining the object sequence according to the sorting result.

[0155] Step S624 , filtering the object sequence according to the object category information corresponding to the graphic and text query information, and determining the target object corresponding to the graphic and text query information according to the filtering result.

[0156] In summary, in order to accurately meet the user's query needs, the user will be provided with a mixed image and text query function. After obtaining the image and text query information input by the user, the information attribute type corresponding to the image and text query information can be determined first. The priority of the query result can be reflected through the information attribute type. After that, the image query information and text query information in the image and text query information can be determined. At this time, the image and text fusion information can be constructed based on the image query information and the text query information, and the image query and text query information can be fused, so that the fusion information can be combined to recall the object later. On this basis, object recall can be performed for image query information, text query information and image-text fusion information respectively, so as to obtain image recall objects, text recall objects and image-text recall objects according to the recall results; on this basis, considering that the recall objects correspond to different query information, the correlations between them and the image-text query information are different, so in order to accurately hit the query intent of the image-text query information, the image recall objects, text recall objects and image-text recall objects can be sorted according to the information attribute type, so as to determine the target objects corresponding to the image-text query information according to the sorting results; when processing object queries, the user's query needs can be hit more accurately, so as to feedback more accurate object query results to the user for user use.

[0157] Corresponding to the above method embodiment, this specification also provides an information processing device embodiment, Figure 7 FIG. 1 shows a schematic diagram of the structure of an information processing device provided by an embodiment of the present specification. Figure 7 As shown, the device comprises:

[0158] The acquisition module 702 is configured to acquire the image and text query information and determine the information attribute type corresponding to the image and text query information;

[0159] The determination module 704 is configured to determine the image query information and the text query information in the image and text query information, and construct the image and text fusion information according to the image query information and the text query information;

[0160] The recall module 706 is configured to perform object recall for the image query information, the text query information and the image-text fusion information respectively, and obtain image recall objects, text recall objects and image-text recall objects;

[0161] The sorting module 708 is configured to sort the image recall objects, the text recall objects and the graphic and text recall objects according to the information attribute type, and determine the target object corresponding to the graphic and text query information according to the sorting result.

[0162] In an optional embodiment, the acquisition module 702 is further configured to:

[0163] The image query information and text query information submitted by the user are received, and object recognition is performed on the image query information; the associated object is determined according to the object recognition result, and the image and text query information is constructed based on the associated object and the text query information.

[0164] In an optional embodiment, the acquisition module 702 is further configured to:

[0165] The associated object and the text query information are input into a multimodal information processing model; object attribute information corresponding to the associated object is extracted through the multimodal information processing model, and graphic and text query information is constructed based on the object attribute information and the text query information.

[0166] In an optional embodiment, the determining module 704 is further configured to:

[0167] The image query information and the text query information are input into a graph-text representation model; the image query features corresponding to the image query information and the text query features corresponding to the text query information are extracted through the graph-text representation model; feature fusion is performed on the image query features and the text query features to obtain graph-text fusion features, and the graph-text fusion features are used as the graph-text fusion information.

[0168] In an optional embodiment, the recall module 706 is further configured to:

[0169] Determine the candidate object features corresponding to the candidate objects in the candidate object set, and input the candidate object features and the image-text fusion features into a multimodal object recall model; calculate the feature similarity between the image-text fusion features and the candidate object features through the multimodal object recall model; screen the target candidate objects in the candidate object set according to the feature similarity, and use the target candidate objects as image-text recall objects.

[0170] In an optional embodiment, the acquisition module 702 is further configured to:

[0171] Determine the first object information corresponding to the image query information according to the object recognition result, and perform entity extraction on the text query information to determine the second object information corresponding to the text query information; detect whether there is an attribute conflict between the image query information and the text query information according to the first object information and the second object information; if so, determine that the information attribute type corresponding to the image and text query information is an image-text conflict attribute type; if not, determine that the information attribute type corresponding to the image and text query information is an image-text non-conflict attribute type.

[0172] In an optional embodiment, the sorting module 708 is further configured to:

[0173] According to a preset sorting strategy, the image recall score corresponding to the image recall object, the text recall score corresponding to the text recall object, and the picture-text recall score corresponding to the picture-text recall object are determined; based on the information attribute type, the image recall score, the text recall score, and the picture-text recall score are updated to obtain a target image recall score, a target text recall score, and a target picture-text recall score; based on the target image recall score, the target text recall score, and the target picture-text recall score, the image recall object, the text recall object, and the picture-text recall object are sorted.

[0174] In an optional embodiment, the device further comprises:

[0175] A category prediction module is configured to input the image-text query information into an object category prediction model for processing, and obtain object category information corresponding to the image-text query information;

[0176] The sorting module 708 is further configured to:

[0177] Determine an object sequence according to the sorting result; filter the object sequence according to the object category information, and determine a target object corresponding to the graphic query information according to the filtering result.

[0178] In an optional embodiment, the acquisition module 702 is further configured to:

[0179] When it is determined according to the recognition result that the image query information contains multiple candidate associated objects, selection items corresponding to the multiple candidate associated objects are fed back to the user; a selection request submitted by the user for the selection item is received, and an associated object is determined from the multiple candidate associated objects based on the selection request.

[0180] The information processing device provided in this embodiment provides a mixed image and text query function to the user in order to accurately meet the query needs of the user. After obtaining the image and text query information input by the user, the information attribute type corresponding to the image and text query information can be determined first, and the priority of the query result can be reflected through the information attribute type. After that, the image query information and text query information in the image and text query information can be determined. At this time, the image and text fusion information can be constructed according to the image query information and the text query information, and the image query and text query information can be fused, so that the fusion information can be combined to recall the object later. On this basis, object recall can be performed for image query information, text query information and image-text fusion information respectively, so as to obtain image recall objects, text recall objects and image-text recall objects according to the recall results; on this basis, considering that the recall objects correspond to different query information, the correlations between them and the image-text query information are different, so in order to accurately hit the query intent of the image-text query information, the image recall objects, text recall objects and image-text recall objects can be sorted according to the information attribute type, so as to determine the target objects corresponding to the image-text query information according to the sorting results; when processing object queries, the user's query needs can be hit more accurately, so as to feedback more accurate object query results to the user for user use.

[0181] The above is a schematic scheme of an information processing device of this embodiment. It should be noted that the technical scheme of the information processing device and the technical scheme of the above information processing method belong to the same concept, and the details not described in detail in the technical scheme of the information processing device can be referred to the description of the technical scheme of the above information processing method.

[0182] Corresponding to the above method embodiment, this specification also provides a commodity query device embodiment, Figure 8 FIG. 1 shows a schematic diagram of the structure of a commodity query device provided by an embodiment of this specification. Figure 8 As shown, the device comprises:

[0183] The information acquisition module 802 is configured to acquire the image and text query information and determine the information attribute type corresponding to the image and text query information;

[0184] The information determination module 804 is configured to determine the image query information and the text query information in the image and text query information, and construct the image and text fusion information according to the image query information and the text query information;

[0185] The product recall module 806 is configured to recall products according to the image query information, the text query information and the image-text fusion information, respectively, to obtain image-recalled products, text-recalled products and image-text recalled products;

[0186] The product sorting module 808 is configured to sort the image recalled products, the text recalled products and the graphic and text recalled products according to the information attribute type, and determine the target product corresponding to the graphic and text query information according to the sorting result.

[0187] The above is a schematic scheme of a commodity query device of this embodiment. It should be noted that the technical scheme of the commodity query device and the technical scheme of the above commodity query method belong to the same concept, and the details not described in detail in the technical scheme of the commodity query device can be referred to the description of the technical scheme of the above commodity query method.

[0188] Corresponding to the above method embodiment, this specification also provides another commodity query device embodiment, Fig. 9 FIG. 2 shows a schematic diagram of the structure of another commodity query device provided by an embodiment of the present specification. Fig. 9 As shown, the device comprises:

[0189] The information receiving module 902 is configured to receive the image and text query information submitted by the user through the product query interface, and determine the information attribute type corresponding to the image and text query information;

[0190] The information determination module 904 is configured to determine the image query information and the text query information in the image and text query information, and construct the image and text fusion information according to the image query information and the text query information;

[0191] The product recall module 906 is configured to recall products according to the image query information, the text query information and the image-text fusion information, respectively, to obtain image-recalled products, text-recalled products and image-text recalled products;

[0192] The product sorting module 908 is configured to sort the image recall products, the text recall products and the graphic and text recall products according to the information attribute type, and determine the target product corresponding to the graphic and text query information according to the sorting result;

[0193] The product display module 910 is configured to update the product query interface to a product display interface including the target product, and display the interface to the user.

[0194] The above is a schematic scheme of another commodity query device of this embodiment. It should be noted that the technical scheme of the commodity query device and the technical scheme of the above commodity query method belong to the same concept, and the details not described in detail in the technical scheme of the commodity query device can be referred to the description of the technical scheme of the above commodity query method.

[0195] Fig.10 The block diagram of a computing device 1000 according to an embodiment of the present specification is shown. The components of the computing device 1000 include but are not limited to a memory 1010 and a processor 1020. The processor 1020 is connected to the memory 1010 via a bus 1030, and the database 1050 is used to store data.

[0196] The computing device 1000 also includes an access device 1040 that enables the computing device 1000 to communicate via one or more networks 1060. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1040 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0197] In one embodiment of the present specification, the above components of the computing device 1000 and Fig.10 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.10 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0198] The computing device 1000 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1000 may also be a mobile or stationary server.

[0199] The processor 1020 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned information processing method or product query method.

[0200] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned information processing method or commodity query method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned information processing method or commodity query method.

[0201] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned information processing method or product query method.

[0202] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned information processing method or commodity query method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned information processing method or commodity query method.

[0203] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned information processing method or product query method.

[0204] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the above-mentioned information processing method or commodity query method belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be referred to the description of the technical scheme of the above-mentioned information processing method or commodity query method.

[0205] An embodiment of the present specification further provides a computer program product, including a computer program or instructions, which implement the steps of the above-mentioned information processing method or product query method when executed by a processor.

[0206] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above-mentioned information processing method or commodity query method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above-mentioned information processing method or commodity query method.

[0207] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0208] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0209] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0210] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0211] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. An information processing method, comprising: Acquire image and text query information, and determine the information attribute type corresponding to the image and text query information; Determining image query information and text query information in the image-text query information, and constructing image-text fusion information according to the image query information and the text query information; Recalling objects respectively for the image query information, the text query information and the image-text fusion information to obtain image recall objects, text recall objects and image-text recall objects; The image recall objects, the text recall objects and the graphic and text recall objects are sorted according to the information attribute type, and the target object corresponding to the graphic and text query information is determined according to the sorting result.

2. The information processing method according to claim 1, wherein obtaining the image and text query information comprises: Receiving image query information and text query information submitted by a user, and performing object recognition on the image query information; Determine the associated object according to the object recognition result, and construct the graphic and text query information based on the associated object and the text query information.

3. The information processing method according to claim 2, wherein the step of constructing the image-text query information based on the associated object and the text query information comprises: inputting the associated object and the text query information into a multimodal information processing model; The object attribute information corresponding to the associated object is extracted through the multimodal information processing model, and the graphic and text query information is constructed based on the object attribute information and the text query information.

4. The information processing method according to claim 1, wherein the step of constructing the image-text fusion information according to the image query information and the text query information comprises: Inputting the image query information and the text query information into a picture-text representation model; Extracting image query features corresponding to the image query information and text query features corresponding to the text query information through the image-text representation model; Feature fusion is performed on the image query feature and the text query feature to obtain an image-text fusion feature, and the image-text fusion feature is used as image-text fusion information.

5. According to the information processing method of claim 4, performing object recall on the image-text fusion information to obtain the image-text recall object comprises: Determine candidate object features corresponding to candidate objects in the candidate object set, and input the candidate object features and the image-text fusion features into a multimodal object recall model; Calculating the feature similarity between the image-text fusion feature and the candidate object feature through the multimodal object recall model; A target candidate object is screened from the candidate object set according to the feature similarity, and the target candidate object is used as a picture and text recall object.

6. The information processing method according to claim 2, wherein determining the information attribute type corresponding to the graphic and text query information comprises: Determine first object information corresponding to the image query information according to the object recognition result, and perform entity extraction on the text query information to determine second object information corresponding to the text query information; detecting, according to the first object information and the second object information, whether there is an attribute conflict between the image query information and the text query information; If so, determining that the information attribute type corresponding to the image-text query information is an image-text conflict attribute type; If not, it is determined that the information attribute type corresponding to the image and text query information is an image and text non-conflicting attribute type.

7. The information processing method according to claim 1, wherein the step of sorting the image recall objects, the text recall objects, and the graphic and text recall objects according to the information attribute type comprises: Determine the image recall score corresponding to the image recall object, the text recall score corresponding to the text recall object, and the picture and text recall score corresponding to the picture and text recall object according to a preset sorting strategy; The image recall score, the text recall score and the picture-text recall score are updated based on the information attribute type to obtain a target image recall score, a target text recall score and a target picture-text recall score; The image recall objects, the text recall objects and the image and text recall objects are sorted based on the target image recall score, the target text recall score and the target image and text recall score.

8. The information processing method according to any one of claims 1 to 7, before the step of sorting the image recall objects, the text recall objects and the graphic and text recall objects according to the information attribute type is performed, it also includes: Inputting the image-text query information into an object category prediction model for processing to obtain object category information corresponding to the image-text query information; Wherein, determining the target object corresponding to the graphic and text query information according to the sorting result includes: Determine the object sequence according to the sorting result; The object sequence is filtered according to the object category information, and the target object corresponding to the graphic and text query information is determined according to the filtering result.

9. The information processing method according to claim 2, wherein determining the associated object according to the object recognition result comprises: When it is determined according to the recognition result that the image query information contains multiple candidate associated objects, feeding back selection items corresponding to the multiple candidate associated objects to the user; A selection request submitted by the user for the selection item is received, and an associated object is determined from among the plurality of candidate associated objects based on the selection request.

10. A commodity query method, comprising: Acquire image and text query information, and determine the information attribute type corresponding to the image and text query information; Determining image query information and text query information in the image-text query information, and constructing image-text fusion information according to the image query information and the text query information; Recall products respectively according to the image query information, the text query information and the image-text fusion information to obtain image-recalled products, text-recalled products and image-text-recalled products; The image recalled products, the text recalled products and the picture and text recalled products are sorted according to the information attribute type, and the target products corresponding to the picture and text query information are determined according to the sorting result.

11. A commodity query method, comprising: Receive the image and text query information submitted by the user through the product query interface, and determine the information attribute type corresponding to the image and text query information; Determining image query information and text query information in the image-text query information, and constructing image-text fusion information according to the image query information and the text query information; Recall products respectively according to the image query information, the text query information and the image-text fusion information to obtain image-recalled products, text-recalled products and image-text-recalled products; Sorting the image recalled products, the text recalled products and the picture and text recalled products according to the information attribute type, and determining the target product corresponding to the picture and text query information according to the sorting result; The product query interface is updated to a product display interface including the target product, and is displayed to the user.

12. An information processing device, comprising: An acquisition module is configured to acquire image and text query information and determine an information attribute type corresponding to the image and text query information; A determination module, configured to determine image query information and text query information in the image-text query information, and construct image-text fusion information according to the image query information and the text query information; A recall module is configured to perform object recall for the image query information, the text query information and the image-text fusion information respectively, and obtain an image recall object, a text recall object and an image-text recall object; The sorting module is configured to sort the image recall objects, the text recall objects and the graphic and text recall objects according to the information attribute type, and determine the target object corresponding to the graphic and text query information according to the sorting result.

13. A commodity query device, comprising: An information acquisition module is configured to acquire image and text query information and determine an information attribute type corresponding to the image and text query information; An information determination module is configured to determine image query information and text query information in the image-text query information, and construct image-text fusion information according to the image query information and the text query information; A product recall module is configured to recall products according to the image query information, the text query information and the image-text fusion information, respectively, to obtain image-recalled products, text-recalled products and image-text-recalled products; The product sorting module is configured to sort the image recall products, the text recall products and the picture and text recall products according to the information attribute type, and determine the target product corresponding to the picture and text query information according to the sorting result.

14. A commodity query device, comprising: An information receiving module is configured to receive the image and text query information submitted by the user through the product query interface, and determine the information attribute type corresponding to the image and text query information; An information determination module is configured to determine image query information and text query information in the image-text query information, and construct image-text fusion information according to the image query information and the text query information; A product recall module is configured to recall products according to the image query information, the text query information and the image-text fusion information, respectively, to obtain image-recalled products, text-recalled products and image-text-recalled products; a product sorting module, configured to sort the image recall products, the text recall products and the graphic and text recall products according to the information attribute type, and determine the target product corresponding to the graphic and text query information according to the sorting result; The commodity display module is configured to update the commodity query interface to a commodity display interface including the target commodity, and display the interface to the user.

15. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 11 are implemented.

16. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

17. A computer program product, comprising a computer program or instructions, which implement the steps of the method according to any one of claims 1 to 11 when executed by a processor.

Images