Image recognition method based on visual language, controller, robot and medium

Abstract

The embodiment of the application provides a kind of image recognition method based on visual language, controller, robot and medium, it is related to artificial intelligence technical field, it is applicable to the field of financial technology and medical health field.The method comprises: obtaining the article use record information of target user;Based on article use record information generates auxiliary find knowledge graph;The graph coding is carried out to auxiliary find knowledge graph, and graph structured semantic feature is obtained;Current scene picture is obtained by collecting the scene where target user is located, and the visual coding is carried out to current scene picture, and current picture feature is obtained;Attention fusion processing is carried out to graph structured semantic feature and current picture feature, and graph picture joint feature is obtained;Acquire the find natural language instruction containing target article description information, determine article search condition according to find natural language instruction and graph picture joint feature.The embodiment of the application can improve the accuracy of article search in specific scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and is suitable for the fields of financial technology and medical health, and in particular relates to an image recognition method based on vision language, a controller, a robot and a medium. BACKGROUND

[0002] A vision language model (VLM) is an artificial intelligence model combining computer vision and natural language processing technology, which is widely used in image recognition, indoor navigation and other fields. For example, in the elderly care scene of the medical health scene, when the elderly issues an elderly article finding instruction (such as "find my hearing aid"), the instruction and a photo of the indoor environment (such as a bedroom or a nursing home room) where the elderly is located can be input into the vision language model together to identify whether the target article exists in the photo, thereby providing indoor navigation services for the elderly. For another example, in the bank business handling scene of the financial technology scene, when the user issues a financial article finding instruction (such as "find my bank card"), the instruction and a picture in the bank hall where the user is located can be input into the vision language model to find the article.

[0003] However, the current vision language model is difficult to accurately identify the required article of a specific user group (such as the elderly) from the image. Specifically, the current vision language model mainly faces general life scenes and is difficult to understand the semantic association between the article finding instruction and the specific scene image (such as the photo of the environment where the elderly is located), thereby being difficult to accurately identify and find the article in the image.

[0004] Therefore, how to improve the accuracy of article finding in a specific scene has become a technical problem to be solved. SUMMARY

[0005] The main purpose of the embodiments of the present application is to provide an image recognition method based on vision language, a controller, a robot and a medium, which aims to improve the accuracy of article finding in a specific scene.

[0006] To achieve the above-mentioned purpose, a first aspect of the embodiments of the present application provides an image recognition method based on vision language, which comprises:

[0007] Obtaining article use record information of a target user; wherein the article use record information comprises a target article name, an article location and a user name of the target user;

[0008] generating a knowledge graph based on the item use record information to obtain an auxiliary item searching knowledge graph; wherein the auxiliary item searching knowledge graph comprises at least two element nodes and the relationship between the element nodes, and each of the element nodes corresponds to one of the target item name, the item location, and the user name;

[0009] performing graph coding on the auxiliary item searching knowledge graph to obtain graph structured semantic features;

[0010] performing image acquisition on a scene where the target user is located to obtain a current scene picture, and performing visual coding on the current scene picture to obtain a current picture feature;

[0011] performing attention fusion processing on the graph structured semantic features and the current picture feature to obtain graph-picture joint features;

[0012] obtaining an item searching natural language instruction containing target item description information, and performing item searching based on the item searching natural language instruction and the graph-picture joint features to obtain an item searching result; wherein the item searching result represents whether the target item described by the target item description information exists in the current scene picture.

[0013] In some embodiments, the item use record information further comprises a use time and an item function;

[0014] The generating a knowledge graph based on the item use record information to obtain an auxiliary item searching knowledge graph comprises:

[0015] generating an item element node based on the target item name to obtain an item element node;

[0016] generating a function element node based on the item function to obtain a function element node, and creating a first edge connecting the function element node and the item element node;

[0017] generating a user element node based on the user name to obtain a user element node, and creating a second edge connecting the user element node and the function element node;

[0018] generating a location element node based on the item location to obtain a location element node, and creating a third edge connecting the location element node and the user element node;

[0019] generating a time element node based on the use time to obtain a time element node, and creating a fourth edge connecting the time element node and the location element node to generate the auxiliary item searching knowledge graph.

[0020] In some embodiments, the location element node is at least two;

[0021] After the time element node is generated based on the use time, and the fourth edge connecting the time element node and the location element node is created, the method further comprises:

[0022] In at least two location element nodes connected by the same article element node, the number of location element nodes with the same article location is counted to obtain the location occurrence frequency of each article location;

[0023] The article location with the maximum location occurrence frequency is determined as the target article location;

[0024] According to the target article location, the auxiliary lost-and-found knowledge graph is updated.

[0025] In some embodiments, the article searching according to the lost-and-found natural language instruction and the graph picture joint feature obtains an article searching result, comprising:

[0026] In response to the lost-and-found natural language instruction, the target article description information is subjected to article name recognition to obtain a first article name recognition result;

[0027] If the first article name recognition result indicates that there is no article name in the target article description information, the target article description information is subjected to feature extraction to obtain an article vague description feature;

[0028] According to the article vague description feature and the graph structured semantic feature, a predicted article name is determined, and the predicted article name is subjected to feature extraction to obtain an article clear description feature;

[0029] According to the article clear description feature and the graph picture joint feature, article feature recognition is performed to obtain the article searching result.

[0030] In some embodiments, the graph structured semantic feature comprises at least two graph node features, each of which corresponds to an element node in the auxiliary lost-and-found knowledge graph;

[0031] According to the article vague description feature and the graph structured semantic feature, the predicted article name is determined, comprising:

[0032] According to the article vague description feature and each graph node feature, feature correlation analysis is performed to obtain a node description feature correlation;

[0033] determine the element node corresponding to the node feature with the maximum correlation degree of the node feature of the graph node as a target association node;

[0034] If the target association node is an article element node, determine the target article name corresponding to the target association node as the predicted article name; wherein the article element node is the element node corresponding to the target article name;

[0035] If the target association node is not the article element node, determine the target article name corresponding to the article element node connected to the target association node as the predicted article name.

[0036] In some embodiments, the lost article natural language instruction is a target lost article text instruction;

[0037] The lost article natural language instruction containing target article description information is obtained, including:

[0038] Obtain the lost article voice instruction of the target user;

[0039] Text conversion is performed on the lost article voice instruction to obtain an original lost article text instruction; wherein the original lost article text instruction contains original article description information;

[0040] Perform article name recognition on the original article description information to obtain a second article name recognition result;

[0041] If the second article name recognition result indicates that there is an article name in the original article description information, determine the article name in the original article description information as an original article name;

[0042] According to the association between the original article name and the article name description information in the preset article name knowledge database, text instruction rewriting is performed on the original lost article text instruction to obtain the target lost article text instruction.

[0043] In some embodiments, the article name description information includes an article alias and a standard article name of the same article;

[0044] According to the association between the original article name and the article name description information in the preset article name knowledge database, text instruction rewriting is performed on the original lost article text instruction to obtain the target lost article text instruction, including:

[0045] Perform text semantic similarity analysis according to the original article name and the article alias to obtain an article name similarity;

[0046] The standard article name corresponding to the article alias with the greatest article name similarity and a preset article searching text instruction template are textually fused to generate the target article searching text instruction.

[0047] To achieve the above object, a second aspect of the embodiments of the present application provides a controller, comprising a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0048] To achieve the above object, a third aspect of the embodiments of the present application provides a robot, comprising the controller of the second aspect.

[0049] To achieve the above object, a fourth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0050] The image recognition method based on visual language, the controller, the robot and the medium provided by the present application obtain the article use record information of the target user, and generate an auxiliary article searching knowledge graph based on the article use record information. The auxiliary article searching knowledge graph can represent the relationship between the target user (such as an elderly person) and the articles used by the target user, such as the relationship between the target article name, the article location and the user name, so as to provide sufficient background knowledge for subsequent article searching. The auxiliary article searching knowledge graph is coded to obtain a graph structured semantic feature, and a current scene picture obtained by collecting the scene where the target user is located is coded to obtain a current picture feature. Then, instead of directly searching for the article according to the image of the scene where the target user is located, the graph structured semantic feature and the current picture feature are fused by an attention mechanism to form a graph picture joint feature, so as to form a joint representation containing user habit knowledge and real-time scene information, and then the graph picture joint feature and the article searching natural language instruction are combined to search for the article. In this way, based on the associated information contained in the auxiliary article searching knowledge graph, the description of the article (i.e., the target article description information) in the article searching natural language instruction and the semantic association between the target article description information and the image of the scene where the target user is located can be more accurately understood, the semantic association between the target article description information and the personalized behavior pattern of the user in a specific scene is strengthened, the semantic understanding error of a general visual language model in a specific user group scene due to the lack of personalized background knowledge is reduced, and the accuracy and reliability of article searching in a specific scene where the target user is located are improved. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a flowchart of the image recognition method based on visual language provided by the embodiments of the present application;

[0052] Figure 2 is a specific implementation schematic diagram of the image recognition method based on visual language provided by another embodiment of the present application;

[0053] Figure 3 is Figure 1 the flowchart of step 102 in

[0054] Figure 4 is a flowchart of the image recognition method based on visual language provided by another embodiment of the present application;

[0055] Figure 5 is a flowchart of step 106 in Figure 1

[0056] Figure 6 is Figure 5 the flowchart of step 403 in

[0057] Figure 7 is a flowchart of step 106 in Figure 1

[0058] Figure 8 is Figure 7 the flowchart of step 605 in

[0059] Figure 9 is a hardware structure schematic diagram of the controller provided by the embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0061] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0063] First, the terms involved in the present application are analyzed:​​

[0064] Artificial Intelligence (AI): is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; Artificial intelligence is a branch of computer science, and artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems, etc. Artificial intelligence can be a simulation of human consciousness and thinking information process. Artificial intelligence can also be a theory, method, technology and application system that uses digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc. The present application can acquire and process related data based on artificial intelligence technology.

[0065] Vision-Language Models (VLM): is an artificial intelligence model that combines computer vision and natural language processing technology. Vision-Language Models combine visual and linguistic modalities, and can learn from images and text to solve cross-modal tasks between images and text. Vision-Language Models are widely used in image recognition, indoor navigation and other fields.

[0066] Knowledge Graph: is a semantic network based on graph structure. Knowledge Graph is used to describe knowledge resources and their mutual relationships. Knowledge Graph includes multiple nodes and edges connecting nodes. Nodes represent entities (such as people, places), concepts or attributes, and edges represent relationships between nodes.

[0067] The image recognition method based on vision language provided by the embodiments of the present application, the controller, the robot and the medium are specifically explained through the following embodiments. First, the image recognition method based on vision language in the embodiments of the present application is described.

[0068] It should be noted that in various specific embodiments of the present application, when relevant processing needs to be performed on data related to the identity or characteristics of the user, such as user information, user behavior data, user history data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the user's separate permission or separate consent will be obtained through a pop-up window or a jump to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for the normal operation of the embodiments of the present application will be obtained.

[0069] Figure 1 is an optional flowchart of the image recognition method based on visual language provided by the embodiments of the present application, Figure 1 The method in can include but is not limited to steps 101-106.

[0070] Step 101, obtaining the item use record information of the target user; wherein the item use record information includes the target item name, the item location, and the user name of the target user;

[0071] Step 102, generating a knowledge graph based on the item use record information to obtain an auxiliary lost-and-found knowledge graph; wherein the auxiliary lost-and-found knowledge graph includes at least two element nodes and the relationship between the element nodes, and each element node corresponds to one of the target item name, the item location, and the user name;

[0072] Step 103, graphically encoding the auxiliary lost-and-found knowledge graph to obtain graph structured semantic features;

[0073] Step 104, image acquisition of the scene where the target user is located to obtain a current scene picture, and visual encoding of the current scene picture to obtain current picture features;

[0074] Step 105, attention fusion processing of the graph structured semantic features and the current picture features to obtain graph-picture joint features;

[0075] Step 106, obtaining a lost-and-found natural language instruction containing target item description information, and performing item searching according to the lost-and-found natural language instruction and the graph-picture joint features to obtain an item searching result; wherein the item searching result represents the presence or absence of the target item described by the target item description information in the current scene picture.

[0076] The beneficial effects of the embodiments of the present application include but are not limited to: obtaining the item use record information of the target user, and generating an auxiliary lost-and-found knowledge graph based on the item use record information, the auxiliary lost-and-found knowledge graph being capable of representing the relationship between the target user (such as an elderly person) and the items used by the target user, such as the relationship between the target item name, the item location, and the user name, so as to provide sufficient background knowledge for subsequent item searching. The auxiliary lost-and-found knowledge graph is coded to obtain a graph structured semantic feature, and the scene in which the target user is located is captured to obtain a current scene picture (i.e., a current scene picture). Then, instead of directly searching for the item according to the image of the scene in which the target user is located, the graph structured semantic feature and the current picture feature are first fused by an attention mechanism to form a graph picture joint feature, so as to form a joint representation containing user habit knowledge and real-time scene information, and then the graph picture joint feature and the lost-and-found natural language instruction are combined to search for the item. In this way, based on the associated information contained in the auxiliary lost-and-found knowledge graph, the description of the item (i.e., the target item description information) in the lost-and-found natural language instruction and the semantic association between the target item description information and the image of the scene in which the target user is located can be more accurately understood, the semantic association between the target item description information and the user's personalized behavior pattern in a specific scene is strengthened, thereby reducing the semantic understanding error of a general visual language model in a specific user group scene due to the lack of personalized background knowledge, and improving the accuracy and reliability of the item searching in the specific scene in which the target user is located.

[0077] In step 101 of some embodiments, the item use record information refers to the record information formed by the target user in the process of using the target item. For example, in the elderly care auxiliary scene of the medical health scene, the target user can be an elderly person (also referred to as an elderly user), and the target item can be an elderly care item. In this case, the item use record information is the record of the elderly person using the elderly care item, such as the record information of the elderly person using a walking stick. Specifically, the item use record information includes the target item name, the item location, and the user name of the target user. The item location can be the common location of the target item. The user name can be used to distinguish and identify the identity of the user.

[0078] In some embodiments, the target item name is the standard item name of the target item. For example, in the elderly care auxiliary scene of the medical health scene, the target item name can include any one of the following names: a hearing aid, a walking stick, a blood pressure-lowering drug, a blood pressure meter, etc. The target item name can also include the names of other drugs, health monitoring devices, or elderly care aids, without being limited thereto. For another example, in the bank business handling scene of the financial technology scene, the target item name can include any one of the following names: a bank card, an ID card, a calculator, a seal, etc.

[0079] In some embodiments, taking the elderly care auxiliary scenario of the medical health scenario as an example, the article use record information of the target user can be acquired from multiple elderly care record source channels such as a nursing home, a family care environment, a health instrument catalog, and an old person behavior record. For example, the name, location, and other information of the article in the environment (such as a nursing home or a family care environment) where the old user is located can be collected, and the article use record information can be generated by combining the name information of the old user. In another embodiment, the article use record information of the target user can also be acquired by other means, which is not limited thereto.

[0080] In step 102 of some embodiments, the auxiliary article searching knowledge graph refers to a knowledge graph used for assisting the article searching task. It should be noted that the auxiliary article searching knowledge graph includes at least two element nodes and the relationship between the element nodes. Specifically, the relationship between the element nodes can be an edge connecting any two element nodes. Each element node corresponds to one article-related data of the article use record information, such as any one of the target article name, the article location, and the user name. For example, the data corresponding to the multiple element nodes in the auxiliary article searching knowledge graph can include “walking stick” (target article name), “bedroom doorway” (article location), and “Zhang Nai Nai” (user name).

[0081] In some embodiments, the data format of the data corresponding to each element node can include text, for example, the target article name, the article location, and the user name are all text data. In another embodiment, the data format of the data corresponding to the element node can also include an image. For example, the element node can correspond to the shape image of the target article, such as the standard image of “walking stick”, to support visual alignment. For another example, the element node can also correspond to the link of the shape image of the target article, such as a URL (Uniform Resource Locator), to reduce the storage resources occupied by the knowledge graph, and thus enable the image recognition method to run on a resource-limited embedded device.

[0082] In some embodiments, the auxiliary article searching knowledge graph can be continuously expanded and updated according to new data, such as new article use record information, so as to adapt to the habits of different user groups in different specific scenarios and improve the individualization degree of the image recognition method.

[0083] In step 103 of some embodiments, the graph structured semantic feature refers to the feature of the auxiliary article searching knowledge graph after coding. For example, as shown in FIG. 3, the graph structured semantic feature of the auxiliary article searching knowledge graph can include the target article name, the article location, and the user name. Figure 2As shown, the atlas (i.e., the auxiliary search knowledge atlas) can be encoded by an atlas encoder. Specifically, the atlas encoder can include an atlas encoder of any one of a Graph Convolutional Network (GCN) model, a Graph Attention Network (GAT) model. In another embodiment, the atlas structured semantic feature can also be encoded in other manners, such as inputting the text of the auxiliary search knowledge atlas into a text encoder for encoding, without being limited thereto.

[0084] In some embodiments, specifically, the atlas structured semantic feature can be a sequence of features including a plurality of element nodes.

[0085] In step 104 of some embodiments, the current scene picture is an image collected from a scene where the target user is located. For example, in the medical health scene of the elderly care assistance scene, the scene where the elderly user (i.e., the target user) is located can include a hospital, a nursing home, a family residence (such as a bedroom), and the like. The current scene picture can be a video frame in a video taken in the elderly care scene where the elderly user is located.

[0086] In some embodiments, it should be noted that the current picture feature is a feature encoded from the current scene picture. For example, as shown, Figure 2 The current scene picture can be visually encoded by a visual encoder. Specifically, the visual encoder can be a visual language model (VLM), such as a visual encoder (also referred to as an image encoder) of a CLIP-ViT model.

[0087] In some embodiments, specifically, the current picture feature corresponding to the current scene picture can be a sequence including block picture features corresponding to a plurality of image blocks in the current scene picture. For example, the current scene picture can be image block segmented to obtain at least two image blocks, and then image feature extraction is performed on each image block to obtain a block picture feature of each image block.

[0088] In step 105 of some embodiments, it should be noted that the atlas picture joint feature is a feature vector fused from the atlas structured semantic feature and the current picture feature. For example, as shown, Figure 2 The atlas structured semantic feature and the current picture feature can be attentionally fused by a domain alignment adapter to obtain the atlas picture joint feature. Specifically, the domain alignment adapter can be a model based on an attention mechanism (such as a cross-attention mechanism), such as a Transformer model.

[0089] In some embodiments, a query can be generated according to the graph structured semantic features, a key and a value can be respectively generated according to the current picture features, and attention calculation can be performed according to the query, the key and the value to obtain graph picture joint features. The graph picture joint features are used to indicate the relevance between each element node in the graph and each local region of the image (i.e., the current scene picture), so as to be able to focus on the image region related to the graph in the current scene picture.

[0090] In step 106 of some embodiments, for example, as shown in FIG. 6, the text (i.e., the object searching natural language instruction) and the graph picture joint features can be subjected to object searching by a visual language model (VLM) to output an object searching result. Figure 2

[0091] In some embodiments, the format of the object searching natural language instruction can be a text format. The target object description information is text information used to describe the target object in the object searching natural language instruction. It should be noted that the target object description information can include any one or more of the following information: the name, function, use time, location, etc. of the object. For example, assuming that the object searching natural language instruction C1 is “where is the thing that I often use to amplify sound”, the target object description information D1 therein is “hearing aid”. For another example, assuming that the object searching natural language instruction C2 is “find the blood pressure lowering medicine tablets that I take every morning”, the target object description information D2 therein is “blood pressure lowering medicine tablets taken every morning”.

[0092] In some embodiments, prompt information such as a voice prompt or a text prompt can be generated according to the object searching result. For example, in the elderly care scene of the medical health scene, assuming that the target object is a walking stick and the object searching result indicates that the walking stick exists in the current scene picture, prompt information “the walking stick is placed at the door of the bedroom” can be generated according to the specific location of the target object in the current scene picture (e.g., the door of the bedroom). Assuming that the object searching result indicates that the target object does not exist in the current scene picture, prompt information “the target object is not seen in the current picture” can be generated.

[0093] In some embodiments, the visual language based image recognition method can be applied to a robot such as an object searching navigation robot. When the object searching navigation robot is located in a scene where a target user is located, a current scene picture can be obtained by a camera in the object searching navigation robot to identify whether a target object (e.g., a walking stick) exists in the current scene picture, and then object navigation can be performed based on the object searching result. For example, if the object searching result indicates that the target object described by the target object description information exists in the current scene picture, a navigation voice can be played to assist the target user in searching for the object. For another example, if the target object does not exist, the object searching navigation robot can play a voice prompt that the object is not identified. ​

[0094] In some embodiments, it should be noted that the visual language model (VLM) is mainly used for multi-modal tasks such as indoor navigation, auxiliary dialogue, image question answering and the like. However, the current visual language model generally relies on large-scale general sample data in the Internet for training, and the visual objects and language descriptions involved in these sample data are mostly related to daily general life scenes, so that the trained visual language model lacks scene adaptation capability for specific groups and specific tasks. For example, in the field of elderly care, especially in the process of robots providing personalized services for the elderly, specific language expressions of the elderly (such as lost object instructions) and specific objects used by the elderly in life, such as medicines, walking sticks, hearing aids commonly used by the elderly, etc. are often involved, and the current visual language model is rarely, if at all, optimized for these specific scene tasks (such as object finding and object navigation tasks). Due to the lack of such sample data during training of the visual language model, and the current visual language model is also difficult to understand the semantic association between specific language expressions and the visual information of objects in specific scene images, it is difficult to accurately recognize images. For example, the robot is prone to recognition failure, understanding deviation or action failure when performing target recognition or navigation tasks, which seriously affects the usability and reliability of the robot in the elderly care scene.

[0095] Based on this, the embodiments of the present application are targeted at specific scenes such as the elderly care scene, and are optimized for domain adaptation and multi-modal alignment to solve the problem of insufficient recognition ability of the image recognition method based on the general VLM model in specific scenes. Specifically, by combining a knowledge graph on the basis of an image, the recognition accuracy of the image recognition method for uncommon objects (such as hearing aids, specific medicines, etc.) in specific scenes is significantly improved, and the response capability of the robot to lost object instructions involving such objects is improved.

[0096] In some embodiments, the model training of the domain perception adapter (see the detailed description of step 105) can be performed according to the object finding situation and the preset standard object existence situation, such as the position of the object in the image. The model training can be performed through a contrast loss function, or can be performed through other ways, without being limited thereto.

[0097] Please refer to Figure 3 In some embodiments, the object use record information further includes a use time and an object function;

[0098] Step 102 can include, but is not limited to, steps 201 to 205:

[0099] Step 201, generating an object element node based on a target object name to obtain an object element node;

[0100] In step 202, a function element node is generated based on the function of the item, a function element node is obtained, and a first edge connecting the function element node and the item element node is created.

[0101] In step 203, a user element node is generated based on the user name, a user element node is obtained, and a second edge connecting the user element node and the function element node is created.

[0102] In step 204, a location element node is generated based on the item location, a location element node is obtained, and a third edge connecting the location element node and the user element node is created.

[0103] In step 205, a time element node is generated based on the use time, a time element node is obtained, and a fourth edge connecting the time element node and the location element node is created, to generate an auxiliary item searching knowledge graph.

[0104] The embodiment has the advantages that the corresponding element nodes are generated based on the item name, the item function, the user name, the item location, and the use time, the item use is associated by the first edge between the item element node and the function element node, the user use behavior is associated by the second edge between the function element node and the user element node, the location of the used item is associated by the third edge between the user element node and the location element node, and the time rule of the item use is associated by the fourth edge between the location element node and the time element node. Thus, the auxiliary item searching knowledge graph can be established based on the multi-dimensional information of the item and the association thereof, sufficient background knowledge support is provided for subsequent item searching, the accuracy and reliability of the item searching in a specific scenario are improved, and the individualized item searching needs of the user (i.e., the target user) are better met.

[0105] In step 201 of some embodiments, the item element node is an element node (referred to as a node) corresponding to the target item name. For example, if the target item name is a walking stick, the item element node can be a "walking stick" node.

[0106] In step 202 of some embodiments, the function element node is an element node corresponding to the item function. For example, if the item function is to assist walking, the function element node can be a "assist walking" node. It should be noted that the first edge is an edge used to connect the function element node and the item element node.

[0107] In step 203 of some embodiments, the user element node is an element node corresponding to the user name. The second edge is an edge used to connect the user element node and the function element node.

[0108] In step 204 of some embodiments, the location element node is an element node corresponding to the location of the item. For example, if the location of the item is the door of the bedroom, the functional element node can be the "door of the bedroom" node. It should be noted that the third edge is an edge for connecting the location element node and the user element node.

[0109] In step 205 of some embodiments, the time element node is an element node corresponding to the time of use. For example, if the time of use is before leaving the house in the morning, the time element node can be the "before leaving the house in the morning" node. It should be noted that the fourth edge is an edge for connecting the time element node and the location element node.

[0110] In some embodiments, it should be noted that the at least two element nodes of the auxiliary lost-and-found knowledge graph include an item element node, a functional element node, a user element node, a location element node, and a time element node; the relationship between the element nodes includes the first edge, the second edge, the third edge, and the fourth edge. For example, the auxiliary lost-and-found knowledge graph can include: "walking stick" (target item name) - "assisting walking" (item function) - "Zhang Nai Nai" (user name) - "door of the bedroom" (item location) - "before leaving the house in the morning" (time of use).

[0111] Please refer to Figure 4 In some embodiments, the location element node is at least two;

[0112] After step 205, the visual language-based image recognition method can further include, but is not limited to, steps 301 to 303:

[0113] Step 301, among the at least two location element nodes connected to the same item element node, the number of location element nodes with the same item location is counted to obtain the location occurrence frequency of each item location;

[0114] Step 302, the item location with the maximum location occurrence frequency is determined as the target item location;

[0115] Step 303, updating the auxiliary lost-and-found knowledge graph according to the target item location.

[0116] The embodiment has the advantages that the position occurrence frequency of each article position is obtained by counting the number of position element nodes with the same article position among the plurality of position element nodes connected to the same article element node, and the article position with the maximum position occurrence frequency is determined as the target article position, so that the common position of the same article can be counted, and the common position is helpful for subsequent article searching. Then, the auxiliary article searching knowledge graph is updated based on the target article position, so that the graph structure can be dynamically optimized, for example, the low-frequency position nodes, that is, the position element nodes corresponding to the positions other than the target article position, can be removed, so as to reduce the interference of low-frequency position information on article searching, reduce the misjudgment probability caused by historical position data redundancy, and thus improve the reliability of article searching.

[0117] In step 301 of some embodiments, the at least two position element nodes connected to the same article element node can correspond to the same or different article positions. For example, assuming that the article element node is a walking stick node, among the 20 position element nodes connected to the walking stick node, 17 position element nodes correspond to the first position, such as the bedroom doorway, and 3 position element nodes correspond to the second position, such as the kitchen doorway. Then, the position occurrence frequency of the first position is 17, and the position occurrence frequency of the second position is 3.

[0118] In step 302 of some embodiments, the target article position is the article position with the maximum position occurrence frequency among the article positions. In the above example, the position occurrence frequency (17) of the first position is greater than the position occurrence frequency (3) of the second position, so the target article position is the first position.

[0119] In step 303 of some embodiments, a deletion operation can be performed on the position element nodes other than the position element nodes corresponding to the target article position among the plurality of position element nodes connected to the same article element node in the auxiliary article searching knowledge graph. In the above example, among the 20 position element nodes connected to the walking stick node, the 3 position element nodes corresponding to the second position and the edges connected to these nodes can be deleted, so that only the position element nodes corresponding to the target article position (i.e., the common position) are retained, so as to improve the reliability of article searching.

[0120] Please refer to Figure 5 In some embodiments, the article searching according to the article searching natural language instruction and the graph picture joint features in step 106 to obtain the article searching situation can include but is not limited to steps 401 to 404:

[0121] Step 401, in response to the article searching natural language instruction, performing article name recognition on the target article description information to obtain a first article name recognition situation;

[0122] Step 402, if the first item name identification condition represents that the item name does not exist in the target item description information, feature extraction is performed on the target item description information to obtain item vague description features;

[0123] Step 403, according to the item vague description features and the atlas structured semantic features, a predicted item name is determined, and feature extraction is performed on the predicted item name to obtain item clear description features;

[0124] Step 404, according to the item clear description features and the atlas picture joint features, item feature recognition is performed to obtain an item search condition.

[0125] The embodiment has the advantages that for a lost item scene in which the description information contained in the lost item instruction lacks an item name, the auxiliary lost item knowledge atlas is used to map the vague description in the lost item instruction to an item name. Specifically, when it is identified that the target item description information of the lost item natural language instruction does not contain an item name, features in the target item description information, i.e., item vague description features, are extracted, and the features are combined with atlas structured semantic features to determine a predicted item name, so that based on the item information association paths (such as the association paths of "item-function-time", etc.) contained in the auxiliary lost item knowledge atlas, the vague description (such as function, time, etc.) of the item in the instruction can be mapped to the predicted item name. Then, according to the item clear description features corresponding to the predicted item name and the atlas picture joint features, recognition is performed to obtain an item search condition. This process uses the semantic association capability of the knowledge atlas, so that it can perform vague search based on the vague instruction lacking an item name, effectively solves the problem of incomplete instruction caused by the expression habit of a specific user group (such as the elderly), and improves the accuracy, reliability and compatibility of item search in a specific scenario.

[0126] In step 401 of some embodiments, the first item name identification condition can represent that the target item description information contains or does not contain an item name. For example, assuming that the lost item natural language instruction is "find my walking stick", the target item description information is "walking stick", and the first item name identification condition represents that the target item description information contains an item name. For another example, assuming that the lost item natural language instruction is "find something I often use when going out for a walk", the target item description information is "something often used when going out for a walk", and the first item name identification condition represents that the target item description information does not contain an item name.

[0127] In step 402 of some embodiments, the item vague description features refer to text features obtained by performing feature extraction on the target item description information that does not contain an item name.

[0128] In step 403 of some embodiments, the predicted item name is an item name stored in the auxiliary item-searching knowledge graph corresponding to the graph structured semantic feature, and the predicted item name is similar to the text semantic feature of the item vague description feature. It should be noted that the item clear description feature refers to the text feature obtained by performing feature extraction on the predicted item name. Since the item clear description feature is the feature of the predicted item name, the subsequent item search can be performed based on the feature of the item name, so as to improve the accuracy of the item search in the case where the item-searching instruction is not clear.

[0129] In step 404 of some embodiments, the item feature recognition can be performed on the item clear description feature and the graph picture joint feature by using a visual language model. The item search condition can also be obtained by other manners, which are not limited thereto.

[0130] Please refer to Figure 6 In some embodiments, the graph structured semantic feature includes at least two graph node features, each of which corresponds to an element node in the auxiliary item-searching knowledge graph.

[0131] Step 403 can include, but is not limited to, steps 501 to 504:

[0132] Step 501, performing feature correlation degree analysis according to the item vague description feature and each graph node feature to obtain a node description feature correlation degree;

[0133] Step 502, determining the element node corresponding to the graph node feature with the maximum node description feature correlation degree as a target correlation node;

[0134] Step 503, if the target correlation node is an item element node, determining the target item name corresponding to the target correlation node as the predicted item name; wherein the item element node is an element node corresponding to the target item name.

[0135] Step 504, if the target correlation node is not an item element node, determining the target item name corresponding to the item element node connected to the target correlation node as the predicted item name.

[0136] The embodiment has the advantages that by calculating the association degree between the item fuzzy description feature and each graph node feature, the element node corresponding to the graph node feature with the highest association degree, i.e., the target association node, is screened out, so as to subsequently find the name of the item described by the item fuzzy description feature based on the association between the nodes of the auxiliary item searching knowledge graph. Specifically, if the target association node corresponds to the item name, i.e., the target association node is the item element node, the item name corresponding to the target association node is directly determined as the predicted item name. If the target association node is not the item element node, the item element node connected to the target association node is found, and then the predicted item name is obtained. In this way, the item name can be derived by using the connection relationship between the nodes in the knowledge graph, so that fuzzy searching is realized, the anti-interference capability for non-standardized expression of the user is enhanced, and the accuracy and reliability of item searching are improved.

[0137] In step 501 of some embodiments, the node description feature association degree is the feature association degree between the item fuzzy description feature and each graph node feature. In some embodiments, the node description feature association degree can be a text feature association degree. For example, the node description feature association degree can be obtained by analysis through a natural language model, an attention model, etc.

[0138] In step 502 of some embodiments, the target association node is the element node corresponding to the graph node feature with the maximum node description feature association degree.

[0139] In step 503 of some embodiments, if the target association node corresponds to the item name, i.e., the target association node is the item element node, the item name corresponding to the target association node is directly determined as the predicted item name. For the meaning and role of the item element node, reference can be made to the specific description of step 201 above, which will not be repeated here.

[0140] In step 504 of some embodiments, if the target association node is not the item element node, the item element node connected to the target association node is found, and then the predicted item name is obtained. In this way, the item name can be derived by using the connection relationship between the nodes in the knowledge graph, so that fuzzy searching is realized.

[0141] Please refer to Figure 7 In some embodiments, the item searching natural language instruction is the target item searching text instruction.

[0142] The obtaining of the item searching natural language instruction containing the target item description information in step 106 can include but is not limited to steps 601 to 605:

[0143] In step 601, the item searching voice instruction of the target user is obtained.

[0144] In step 602, text conversion is performed on the lost item voice instruction to obtain an original lost item text instruction; wherein the original lost item text instruction contains original item description information.

[0145] In step 603, item name recognition is performed on the original item description information to obtain a second item name recognition result.

[0146] In step 604, if the second item name recognition result indicates that the original item description information contains an item name, the item name in the original item description information is determined as an original item name.

[0147] In step 605, text instruction rewriting is performed on the original lost item text instruction according to the association between the original item name and item name description information in the preset item name knowledge database to obtain a target lost item text instruction.

[0148] The embodiment has the advantage that the language expression of the target user often has ambiguity and a tendency of colloquialism, and the item name in the language expression may not be the accurate name of the item. To solve the above problem, the original instruction is rewritten to make it standardized. Specifically, the lost item voice instruction of the target user is converted into text, and in the case that the original lost item text instruction contains an item name (i.e., an original item name), text instruction rewriting is performed on the original lost item text instruction according to the association between the original item name and item name description information in the preset item name knowledge database, so as to rewrite the ambiguous instruction into an instruction containing a standard item name. For example, the colloquial expression of the user can be rewritten into a standard item name with the same or similar semantics, so that the instruction text input in the subsequent item finding stage is consistent with the item naming stored in the knowledge graph, thereby improving the ability to process non-standardized instructions and improving the accuracy of item finding in specific scenarios.

[0149] In step 601 of some embodiments, the lost item voice instruction is a voice instruction of a target user. For example, the lost item voice instruction of the target user can be acquired through a voice acquisition module of a robot. Taking the elderly care scene of the medical health scene as an example, the elderly user can say “find my earphone” to the robot, and the robot can acquire the voice to obtain the lost item voice instruction. It should be noted that the language expression of the target user (such as an elderly user) often has ambiguity and a tendency of colloquialism, and the item name in the language expression may not be the accurate name of the item, but an alias of the item, or a colloquial expression caused by personal habits, such as calling “hearing aid” as “earphone”.

[0150] In step 602 of some embodiments, the original lost item text instruction is text converted from the lost item voice instruction. Specifically, the lost item voice instruction can be converted into text by a pre-trained speech recognition model. It should be noted that the original item description information refers to the item description information in the original lost item text instruction. For example, assuming that the original lost item text instruction is "find my commonly used earphone", the original item description information is "commonly used earphone".

[0151] In step 603 of some embodiments, the second item name identification condition can represent the presence or absence of an item name in the original item description information. It should be noted that when the second item name identification condition represents the presence of an item name in the original item description information, the item name in the original item description information can be an item alias. In addition, the item name can also be other types of colloquial, non-standardized item names, such as item names in dialects.

[0152] In step 604 of some embodiments, the original item name is the item name in the original item description information. As in the above example, assuming that the original item description information is "commonly used earphone", the original item name therein is "earphone".

[0153] In step 605 of some embodiments, the item name knowledge database is a database for storing multiple names of the same item, including the alias and the standard name of the item. In some embodiments, the item name knowledge database can include multiple item name description information. It should be noted that each item name description information is used to represent the mapping relationship between the item alias and the standard item name of the same item.

[0154] It should be noted that the target lost item text instruction is a text instruction obtained by revising the original lost item text instruction. The target lost item text instruction contains a standardized item name, that is, the standard item name in the following text. As in the above example, assuming that the original item name is "earphone", the standard item name in the target lost item text instruction can be "hearing aid". In some embodiments, it should be noted that the language expression of the elderly user often has ambiguity and colloquial tendency, such as "take the medicine I eat every day", "find the earphone I usually wear", etc. Based on this, the embodiments of the present application can convert the language expression of the elderly user into a standard instruction recognizable by the robot, replace the colloquial and ambiguous language expression, and let the revised instruction retain the original meaning, improve the naturalness and fault tolerance of the interaction between the robot and the elderly user.

[0155] Please refer to Figure 8 In some embodiments, the item name description information includes the item alias and the standard item name of the same item.

[0156] Step 605 can include but is not limited to steps 701-702:

[0157] In step 701, text semantic similarity analysis is performed on the original item name and the item alias to obtain an item name similarity.

[0158] In step 702, text fusion is performed on the standard item name corresponding to the item alias with the maximum item name similarity and a preset item searching text instruction template to generate a target item searching text instruction.

[0159] The embodiment has the advantage that, by analyzing the text semantic similarity between the original item name and the item alias, the item name similarity is obtained, the standard item name corresponding to the item alias with the maximum item name similarity is screened out, and thus the standard item name of the item is found based on the original item name in the instruction. Then, the standard item name is embedded into the item searching text instruction template to generate a target item searching text instruction, so as to rewrite the colloquial instruction containing the item alias into an instruction containing the standard item name, reduce the influence of entity reference ambiguity caused by the expression difference of the user group, and further improve the accuracy of item searching in a specific scenario.

[0160] In step 701 of some embodiments, the item name similarity refers to the text semantic similarity between the original item name and the item alias.

[0161] In step 702 of some embodiments, it should be noted that the item name similarity can be used to analyze which item the original item name is an alias of. For example, it is assumed that the original item name is "ear machine", and the item name description information 1 includes: "hearing aid" (standard item name) - "ear machine" (item alias); and the item name description information 2 includes: "nitroglycerin tablet" (standard item name) - "heart pain medicine" (item alias). Then, among the multiple item name description information, the similarity between the item alias "ear machine" in the item name description information 1 and the original item name is the largest, and thus the standard item name "hearing aid" corresponding to the item alias "ear machine" is the standard item name obtained after screening, which is a different reference of the same item entity as the original item name.

[0162] The embodiment of the present application also provides a controller, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above-mentioned image recognition method based on visual language when executing the computer program. In some embodiments, the controller can include a control chip in a robot. In another embodiment, the controller can also include other intelligent terminals.

[0163] Please refer to Figure 9 , Figure 9 The hardware structure of the controller of another embodiment is illustrated, which includes:

[0164] The processor 901 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing relevant programs to implement the technical solutions provided by the embodiments of the present application.

[0165] The memory 902 can be implemented by a ROM (Read-Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), etc. The memory 902 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, relevant program codes are stored in the memory 902 and are called and executed by the processor 901 to implement the image recognition method based on visual language according to the embodiments of the present application.

[0166] The input / output interface 903 is configured to realize information input and output.

[0167] The communication interface 904 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0168] The bus 905 is configured to transmit information between various components (for example, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device.

[0169] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are connected to each other through the bus 905 to realize the communication connection between the device.

[0170] The embodiments of the present application further provide a robot, and the robot comprises the controller.

[0171] It should be noted that the image recognition method based on visual language can be applied to a robot. In an embodiment, the robot includes but is not limited to a lost object navigation robot, an elderly care assisting robot, and the like. The robot can also include other functional or type robots, and is not limited thereto.

[0172] The specific implementation of the robot is basically the same as that of the above-mentioned specific embodiments of the image recognition method based on visual language, and will not be repeated here.

[0173] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the image recognition method based on visual language.

[0174] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0175] It should be noted that the non-company software tools or components appearing in the embodiments of the present application are only examples for introduction, and do not represent actual use.

[0176] The embodiments described in the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0177] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures shown, or combine certain steps, or different steps.

[0178] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separated, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0179] Those skilled in the art can understand that all or some steps in the above disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0180] The terms "first", "second", "third", "fourth", and the like in the description of this application and in the claims hereof, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed herein is solely for the convenience of the reader and does not limit the scope of the application. It is also to be understood that the description and examples in this application are intended to cover all possible combinations where any of the several elements can represent one or more elements.

[0181] It should be understood that, in this application, "at least one" means one or more, "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c, can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0182] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above-mentioned units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. The coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0183] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment of the present application.

[0184] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0185] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store programs.

[0186] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and the scope of the rights of the embodiments of the present application is not limited thereto. Any modification, equivalent replacement and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.

Claims

1. A method for image recognition based on visual language, characterized in that, The method comprises: acquiring item use record information of a target user; wherein the item use record information comprises a target item name, an item location, and a user name of the target user; generating a knowledge graph based on the item use record information to obtain an auxiliary item searching knowledge graph; wherein the auxiliary item searching knowledge graph comprises at least two element nodes and the relationships between the element nodes, and each element node corresponds to one of the target item name, the item location, and the user name; performing graph coding on the auxiliary item searching knowledge graph to obtain graph structured semantic features; performing image acquisition on a scene in which the target user is located to obtain a current scene picture, and performing visual coding on the current scene picture to obtain current picture features; performing attention fusion processing on the graph structured semantic features and the current picture features to obtain graph-picture joint features; acquiring an item searching natural language instruction containing target item description information, and performing item searching based on the item searching natural language instruction and the graph-picture joint features to obtain an item searching result; wherein the item searching result represents whether the target item described by the target item description information exists in the current scene picture.

2. The method of claim 1, wherein, The item use record information further comprises a use time and an item function; The generating of the auxiliary item searching knowledge graph based on the item use record information comprises: generating an item element node based on the target item name to obtain an item element node; generating a function element node based on the item function to obtain a function element node, and creating a first edge connecting the function element node and the item element node; generating a user element node based on the user name to obtain a user element node, and creating a second edge connecting the user element node and the function element node; generating a location element node based on the item location to obtain a location element node, and creating a third edge connecting the location element node and the user element node; generating a time element node based on the use time to obtain a time element node, and creating a fourth edge connecting the time element node and the location element node to generate the auxiliary item searching knowledge graph.

3. The method of claim 2, wherein, The location element node is at least two; After the generating of the time element node based on the use time and the creating of the fourth edge connecting the time element node and the location element node, the method further comprises: counting the number of location element nodes with the same item location among at least two location element nodes connected to the same item element node to obtain the location occurrence frequency of each item location; determining the item location with the maximum location occurrence frequency as a target item location; updating the auxiliary item searching knowledge graph based on the target item location.

4. The method according to any one of claims 1 to 3, characterized in that, The item searching according to the item searching natural language instruction and the atlas picture joint feature obtains an item searching condition, and the item searching condition comprises: In response to the item searching natural language instruction, item name recognition is performed on the target item description information to obtain a first item name recognition condition; If the first item name recognition condition indicates that there is no item name in the target item description information, feature extraction is performed on the target item description information to obtain an item vague description feature; According to the item vague description feature and the atlas structured semantic feature, a predicted item name is determined, and feature extraction is performed on the predicted item name to obtain an item clear description feature; According to the item clear description feature and the atlas picture joint feature, item feature recognition is performed to obtain the item searching condition.

5. The method of claim 4, wherein, The atlas structured semantic feature comprises at least two atlas node features, and each atlas node feature corresponds to an element node in the auxiliary item searching knowledge atlas; According to the item vague description feature and each atlas node feature, feature correlation degree analysis is performed to obtain a node description feature correlation degree; The element node corresponding to the atlas node feature with the maximum node description feature correlation degree is determined as a target correlation node; If the target correlation node is an item element node, the target item name corresponding to the target correlation node is determined as the predicted item name; wherein the item element node is the element node corresponding to the target item name; If the target correlation node is not the item element node, the target item name corresponding to the item element node connected to the target correlation node is determined as the predicted item name. The item searching natural language instruction is a target item searching text instruction; 6. The method according to any one of claims 1 to 3, characterized in that, The obtaining of the item searching natural language instruction containing target item description information comprises: Obtaining an item searching voice instruction of the target user; Text conversion is performed on the item searching voice instruction to obtain an original item searching text instruction; wherein the original item searching text instruction contains original item description information; Item name recognition is performed on the original item description information to obtain a second item name recognition condition; If the second item name recognition condition indicates that there is an item name in the original item description information, the item name in the original item description information is determined as an original item name; According to the association between the original item name and item name description information in a preset item name knowledge database, text instruction rewriting is performed on the original item searching text instruction to obtain the target item searching text instruction. The item name description information comprises an item alias and a standard item name of the same item; 7. The method of claim 6, wherein, According to the association between the original item name and item name description information in a preset item name knowledge database, text instruction rewriting is performed on the original item searching text instruction to obtain the target item searching text instruction, comprising: ​ performing text semantic similarity analysis on the original item name and the item alias to obtain an item name similarity; performing text fusion on the standard item name corresponding to the item alias with the greatest item name similarity and a preset item searching text instruction template to generate the target item searching text instruction.

8. A controller characterized by comprising: The controller comprises a memory and a processor, the memory stores a computer program, and the processor implements the method in any one of claims 1 to 7 when executing the computer program.

9. A robot, characterized in that The robot comprises the controller in claim 8.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the method in any one of claims 1 to 7. The computer program is executed by the processor to implement the method in any one of claims 1 to 7.

Images