Item searching method and system based on interaction, electronic equipment and medium

Abstract

The invention discloses an interaction-based article search method and system, electronic equipment and a storage medium, and the method comprises the steps: analyzing a voice search instruction of a user, and determining the name of an article searched by the user; querying in a database according to the name of the article, and determining related information of the article searched by the user, including position information of the article; feedback information is determined according to the article related information and a preset feedback template; responding to the user according to the determined feedback information, and informing the user of the position information of searching the article; by analyzing the user voice instruction, extracting the key information for searching the article and determining the position information of the article according to the key information and the database for feedback response, the user can obtain the position information of the article through voice inquiry, the operation is simple and convenient, and the user experience is improved; the article is searched according to the feedback response information, so that the article searching time of the user is shortened, and the article searching efficiency is improved; the embodiment of the invention can be widely applied to the technical field of data processing.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to an interactive-based item search method, system, electronic device, and medium. Background Art

[0002] In modern families, the types and quantities of various electronic devices, tools, and other items are constantly increasing, making the complexity and difficulty of managing and searching for items in the family higher and higher, wasting a large amount of time, manpower, and material resources, and affecting the living efficiency and quality; traditional item search relies on manual memory or search based on manually recorded information. However, manual memory is prone to omissions and errors as the number of items increases and over time, and manually recorded information also needs to be continuously updated and maintained, resulting in low item search efficiency. Summary of the Invention

[0003] The main objective of the embodiments of the present invention is to provide an interactive-based item search method, system, electronic device, and storage medium, which can improve the search efficiency and enhance the user experience.

[0004] To achieve the above objective, on the one hand, an embodiment of the present invention provides an interactive-based item search method, which includes:

[0005] Performing speech recognition on the received voice query instruction to determine query content information; wherein, the query content information includes the item name and the corresponding description information;

[0006] Querying a database according to the query content information to determine a query result; wherein, the query result includes item information and the corresponding location information;

[0007] Processing according to the query result and a preset feedback template to determine a response message; and making a response according to the response message.

[0008] In some embodiments, the performing speech recognition on the received voice query instruction to determine query content information specifically includes:

[0009] Performing frame processing on the voice query instruction to determine several groups of frame signals, extracting features from the several groups of frame signals to determine a time series feature set;

[0010] Performing time series feature extraction on the time series feature set to determine a feature information set; mapping the feature information set to determine a label probability set; wherein, the feature information set includes time series dependency and context relationship information;

[0011] Decoding the label probability set to determine a query text, and using the query text as the query content information.

[0012] In some embodiments, processing according to the query result and a preset feedback template to determine a response message specifically includes:

[0013] Matching the query result with the preset feedback template to determine feedback text information;

[0014] Performing feature extraction on the feedback text information to determine text feature information; performing modeling based on the text feature information to determine a voice feature data set;

[0015] Performing voice synthesis based on the voice feature data set to determine voice waveform data; using the voice waveform data as the response message.

[0016] In some embodiments, the method further includes:

[0017] Obtaining an initialization instruction, parsing the initialization instruction to determine initialization information; wherein the initialization information includes a working area and boundary information;

[0018] Performing an environmental scan according to the initialization instruction to determine environmental data; performing map modeling based on the environmental data to determine three-dimensional map data; configuring the three-dimensional map data according to the initialization information to determine a system map.

[0019] In some embodiments, the method further includes:

[0020] Parsing a received record instruction to determine item information; wherein the item information includes an item name and item location information;

[0021] Performing data collection according to the record instruction to determine an item image set; performing feature extraction on the item image set to determine an item feature set;

[0022] Generating index information based on the item information and the item feature set, and saving the index information to the database.

[0023] In some embodiments, the method further includes:

[0024] Scanning the current environment according to a preset period to determine a current environment data set; parsing the current environment data set to determine current item information data;

[0025] Comparing the current item information data with the database to determine abnormal item information; generating a request message based on the abnormal item information; wherein the abnormal item information includes item information with a changed location information, and the request message is used for a user to confirm the changed location information;

[0026] Obtain feedback information, and update the database according to the feedback information and the abnormal item information; wherein, the feedback information includes item information for which the user confirms a change in location information.

[0027] In some embodiments, the method further includes:

[0028] Determine a target item according to a user instruction, query the database according to the target item, and determine the location information of the target item;

[0029] Perform path planning according to the target location information, preset storage area information, and system map to determine a storage path;

[0030] Generate a drive instruction according to the storage path, and send the drive instruction to a storage robot, so that the storage robot moves the target item to a preset storage area according to the drive instruction.

[0031] To achieve the above object, another aspect of the embodiments of the present invention provides an interactive-based item search system, including:

[0032] A first module, configured to perform speech recognition on a received voice query instruction to determine query content information; wherein, the query content information includes an item name and corresponding description information;

[0033] A second module, configured to query a database according to the query content information to determine a query result; wherein, the query result includes item information and corresponding location information;

[0034] A third module, configured to process according to the query result and a preset feedback template to determine a response message; and make a response according to the response message.

[0035] To achieve the above object, another aspect of the embodiments of the present application proposes an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the foregoing method is implemented.

[0036] To achieve the above object, another aspect of the embodiments of the present application proposes a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the foregoing method is implemented.

[0037] Implementing the embodiments of the present invention includes the following beneficial effects: This embodiment provides an interactive-based item search method, system, electronic device, and storage medium. This solution parses the user's voice search instruction to determine the name of the item the user needs to search for; queries in the database based on the determined item name to determine the relevant information of the item the user needs to search for, including the location information of the item; then, processes according to the obtained item-related information and a preset feedback template to determine the feedback information; responds to the user based on the determined feedback information to inform the user of the relevant information of the item to be searched for; extracts the key information of the item to be searched by parsing the user's voice instruction, determines the location information of the item based on the key information and the database, and gives a feedback response. The user can obtain the location information of the item by voice inquiry or issuing an instruction, which is easy to operate and improves the user experience; searching for the item according to the information in the feedback response can shorten the time for the user to search for the item and improve the item search efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a schematic flowchart of the steps of an interactive-based item search method provided by an embodiment of the present invention;

[0039] Figure 2 is a schematic flowchart of the steps of determining query content information in an interactive-based item search method provided by an embodiment of the present invention;

[0040] Figure 3 is a schematic flowchart of the steps of determining response information in an interactive-based item search method provided by an embodiment of the present invention;

[0041] Figure 4 is a schematic flowchart of the steps of initializing and constructing a system map in an interactive-based item search method provided by an embodiment of the present invention;

[0042] Figure 5 is a schematic flowchart of the steps of recording item information in an interactive-based item search method provided by an embodiment of the present invention;

[0043] Figure 6 is a schematic flowchart of the steps of updating the database in an interactive-based item search method provided by an embodiment of the present invention;

[0044] Figure 7 is a schematic flowchart of the steps of driving a robot to perform item storage in an interactive-based item search method provided by an embodiment of the present invention;

[0045] Figure 8 is a structural block diagram of an intelligent storage robot in a specific embodiment provided by an embodiment of the present invention;

[0046] Figure 9 It is a schematic flow chart of the steps for storing and managing items in a specific embodiment provided by an embodiment of the present invention;

[0047] Figure 10 It is a block diagram of the structure of an item search system based on interaction provided by an embodiment of the present invention;

[0048] Figure 11 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0049] The following further elaborates on the present invention in detail in conjunction with the accompanying drawings and specific embodiments. For the step numbers in the following embodiments, they are only set for the convenience of elaboration and explanation, and no limitation is imposed on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0050] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0051] In the following description, the terms "first\second\third" are only used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence when allowed, so that the embodiments of the present invention described here can be implemented in an order other than that illustrated or described here.

[0052] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present invention have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs. The terms used in the embodiments of the present invention are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0053] Before further elaborating on the embodiments of the present invention, the nouns and terms involved in the embodiments of the present invention are described. The nouns and terms involved in the embodiments of the present invention are applicable to the following explanations.

[0054] Radio Frequency Identification (RFID): RFID is a type of automatic identification technology that performs non-contact two-way data communication through radio frequency, and reads and writes a recording medium (electronic tag or RF card) using radio frequency, so as to achieve the purpose of identifying the target and exchanging data;

[0055] Natural Language Processing: Natural language processing is an important branch in the field of artificial intelligence, involving the interdisciplinary study of computer science, linguistics, and mathematics. It enables computers to understand and process human language, achieving natural communication between humans and machines;

[0056] Computer Vision: Computer vision is a simulation of biological vision using computers and related devices. By processing the captured images or videos, it obtains the three-dimensional information of the corresponding scene;

[0057] Natural Language Generation: Natural language generation is an important part of natural language processing. It converts structured data or other forms of information into natural language text or speech output, enabling computers to interact and communicate with humans in a natural way.

[0058] In related technologies, daily necessities are managed through manual memory or handwritten records. When it is necessary to find daily necessities, they are searched according to the memory or the information in the handwritten records; or two-dimensional codes or electronic tags using radio frequency technology are assigned to daily necessities for item management. The relevant information of daily items is recorded by scanning the electronic tags or two-dimensional codes, and searched according to the recorded information. However, as time goes by or the quantity and types of daily items increase, the memory may be omitted or incorrect, and the content of the handwritten records needs to be continuously updated and maintained, which is prone to errors and has low search efficiency; while using electronic tags or two-dimensional codes requires additional devices for assistance, which is not highly applicable to the home environment; at the same time, when using electronic tags or two-dimensional codes for item storage management, the items need to be processed in advance, such as pasting two-dimensional codes or attaching electronic tags, which affects the user experience.

[0059] In view of this, the embodiments of the present invention provide an interactive-based item search method, system, electronic device, and storage medium. This solution models the environment in the working area, records the relevant information of daily necessities based on computer vision and sensors, including appearance features, positions, types, names, etc., and saves the recorded relevant information in the database. It parses the user's language instructions through natural language processing technology, extracts the key information of the item to be searched, automatically searches for the corresponding item information in the database according to the key information, generates corresponding voice or text for response through natural language processing technology, and stores the items according to the constructed map; realizes the automatic recording and information search of daily items. The user can obtain the location information of the item only by asking verbally, simplifies the item search process, improves the search efficiency, and enhances the user experience.

[0060] Figure 1 It is an optional flowchart of an item search method based on interaction provided by an embodiment of the present invention. Figure 1 The method in it may include but is not limited to steps S101 to S103.

[0061] Step S101: Perform speech recognition on the received voice query instruction to determine the query content information; wherein, the query content information includes the item name and the corresponding description information.

[0062] Step S102: Query the database according to the query content information to determine the query result; wherein, the query result includes the item information and the corresponding location information.

[0063] Step S103: Process according to the query result and a preset feedback template to determine the response information; and make a response according to the response information.

[0064] In steps S101 to S103 illustrated in the embodiments of the present application, the system collects the user's voice instruction through a microphone. The voice instruction at least includes the item name to be searched, such as "water cup". First, the system uses speech recognition technology to process the collected voice instruction, converting the voice instruction into text data to ensure that the user's query content can be accurately captured even in a noisy environment. Then, the system uses natural language processing technology to parse the converted text data, extracting the key information in the user's voice instruction, for example, the item name to be searched, the context relationship of the instruction, etc. Query the system's database according to the extracted key information to obtain the item data corresponding to the key information, including the item name, category, location information, etc. Based on the queried item data, use natural language generation technology to generate text, convert the queried item data into natural language text that is easy for the user to understand, and then use speech synthesis technology to convert the natural language text into a voice signal. Finally, make a response to the user according to the voice signal, informing the user of the location information of the searched item.

[0065] In step S103 of some embodiments, the queried item information can be converted into voice data through natural language generation technology and speech synthesis technology in sequence, and a response can be made to the user through devices such as a microphone. It can also convert the queried item information into corresponding text information through natural language generation technology and make a response to the user through display devices such as a display screen, and is not limited thereto.

[0066] Please refer to Figure 2 In some embodiments, step S101 may include but is not limited to steps S201 to S203:

[0067] Step S201: Perform frame segmentation on the voice query instruction to determine several groups of frame signals, extract features from the several groups of frame signals, and determine the time series feature set.

[0068] Step S202: Perform time series feature extraction on the time series feature set to determine the feature information set; perform mapping on the feature information set to determine the label probability set; wherein, the feature information set includes time series dependency relationship and context relationship information.

[0069] Step S203: Decode the label probability set to determine the query text, and use the query text as the query content information.

[0070] In step S201 of some embodiments, the system uses speech recognition technology to convert the voice instruction input by the user into text for subsequent item information query; in this embodiment, a deep learning model is used to process the voice instruction input by the user, such as an end-to-end speech recognition network; in order to improve the accuracy of speech conversion, first perform frame segmentation on the user's voice instruction to reduce interference from environmental noise, etc.; then, extract features from the several speech frames obtained by frame segmentation respectively, and extract the time series features of the speech signals in the voice instruction.

[0071] In step S202 of some embodiments, the system performs time series feature extraction on the extracted time series features to determine the trend features of the speech energy in each speech frame and the dependency relationship features between different speech frames, as well as the context relationship features of the entire voice instruction; then, classify according to the extracted speech time series features to determine the character probability distribution corresponding to each speech frame.

[0072] In step S203 of some embodiments, the system performs decoding analysis on the character probability distribution of each speech frame to determine the text characters after conversion of each speech frame, such as "I", "cup", etc.; then, according to the context relationship and time series dependency relationship of the voice instruction, determine the text information after conversion of the voice instruction, and the system performs item information query in the database according to the converted text information.

[0073] Please refer to Figure 3 , in some embodiments, step S103 may include but is not limited to steps S301 to S303:

[0074] Step S301: Match the query result with a preset feedback template to determine the feedback text information.

[0075] Step S302: Extract features from the feedback text information to determine the text feature information; perform modeling according to the text feature information to determine the voice feature data set.

[0076] Step S303: Perform speech synthesis based on the speech feature dataset to determine speech waveform data; use the speech waveform data as the response information.

[0077] In step S301 of some embodiments, the system queries in the database according to the text information after the speech instruction conversion. After determining the corresponding item information, in order to facilitate the user to understand the queried item information, the system converts the corresponding item information into a text or speech signal in natural language; the system matches the queried item information with a preset feedback template, and uses preset characters to supplement the slots of the feedback template to obtain the corresponding feedback text information.

[0078] In step S302 of some embodiments, the system extracts features from the obtained feedback text information and converts them into corresponding linguistic features, including pitch, syllable boundaries, etc.; then, annotates the obtained linguistic features with prosody information to convert the original text information into natural intonation data; finally, the system uses a front-end model to process the obtained natural intonation data to generate acoustic features; exemplarily, uses the Tacotron2 model to convert the obtained natural intonation data into a Mel spectrogram as an intermediate acoustic feature, and the system generates a feedback speech signal according to the obtained acoustic features.

[0079] In step S303 of some embodiments, the system inputs the obtained acoustic features into a speech synthesis model for processing, extracts the dependencies in the acoustic features, and maintains the temporal causal relationship in the acoustic features; then, performs autoregressive generation according to the acoustic features, generates the speech waveform data corresponding to the feedback text information point by point and outputs it as a speech signal; uses the speech signal output by the speech synthesis model as the response information of the system.

[0080] Please refer to Figure 4 , a method for item search based on interaction provided by an embodiment of the present invention may include but is not limited to steps S401 to S402:

[0081] Step S401: Obtain an initialization instruction, parse the initialization instruction to determine initialization information; where the initialization information includes a working area and boundary information.

[0082] Step S402: Perform environmental scanning according to the initialization instruction to determine environmental data; perform map modeling according to the environmental data to determine three-dimensional map data; configure the three-dimensional map data according to the initialization information to determine the system map.

[0083] In step S401 of some embodiments, before the user searches for an item through a voice command, it is necessary to deploy a system or device carrying the item search method proposed in this embodiment in the environment where item management is required; after the system or device is deployed, it is necessary to perform an initialization configuration on the system or device and then enter normal operation; the user can operate through the display screen to input an initialization command, or input a corresponding initialization voice command; after receiving the initialization command, the system parses the initialization command and extracts the working area defined by the user in the command, such as a room, a warehouse, etc., and the corresponding boundary area. For example, the user uses a private space such as a bathroom as a boundary limit. When the system manages or searches for items, it will not cross the boundary and violate the user's privacy.

[0084] In step S402 of some embodiments, after the system determines the initialization information, it scans the environment of the working area set in the initialization through the carried sensor or radar, and collects the environmental data in the working area; performs map modeling based on the collected environmental data to obtain three-dimensional map data; the system can subsequently mark the items that need to be recorded and managed in the constructed three-dimensional map, and the user can intuitively determine the item positions according to the three-dimensional map; the system can also perform path planning based on the constructed three-dimensional map to drive the storage robot to perform storage management on the items in the working area.

[0085] Please refer to Figure 5 , a method for searching for items based on interaction provided by an embodiment of the present invention may include but is not limited to steps S501 to S503:

[0086] Step S501, parse the received voice command to determine the item information; wherein, the item information includes the item name and the item position information;

[0087] Step S502, perform data collection according to the voice command to determine the item image set; extract features from the item image set to determine the item feature set;

[0088] Step S503, generate index information according to the item information and the item feature set, and save the index information to the database.

[0089] In step S501 of some embodiments, after the initialization configuration is completed, the system needs to record the items that need to be managed for subsequent item search according to the user's voice command; the user can control the system to record the items through the corresponding voice command;

[0090] In step S502 of some embodiments, after receiving a voice command, the system processes the voice command through a preset voice recognition algorithm to extract the command content, such as "record the water cup"; the system detects the items in the current environment through sensors and cameras, collects the image data and position information of the water cup, and analyzes the collected image data to extract the appearance features of the water cup.

[0091] In step S503 of some embodiments, the system generates index information corresponding to the item based on the item name in the voice command, the collected appearance features of the item, and the position information of the item, etc., and stores the generated index information in the database for subsequent item search to determine the position information of the item; in this embodiment, the system also collects the corresponding timestamp and stores it in the database accordingly; the system can determine the recording time of the current item index information according to the timestamp, set the corresponding update period, and update the data in the database regularly according to the timestamp to ensure the timeliness and accuracy of the data.

[0092] Please refer to Figure 6 , a method for finding an item based on interaction provided by an embodiment of the present invention may include but is not limited to steps S601 to S603:

[0093] Step S601, scan the current environment according to a preset period to determine the current environment data set; parse the current environment data set to determine the current item information data;

[0094] Step S602, compare the current item information data with the database to determine the abnormal item information; generate a request information according to the abnormal item information; wherein, the abnormal item information includes the item information with a changed position information, and the request information is used for the user to confirm the changed position information;

[0095] Step S603, obtain feedback information, and update the database according to the feedback information and the abnormal item information; wherein, the feedback information includes the item information for which the user confirms the changed position information.

[0096] In step S601 of some embodiments, the user can set an update period in the system, and the system regularly detects the environment of the working area according to the set update period, collects the current environment data, and the system parses the collected environment data to determine the appearance features and corresponding position information of the items in the working area;

[0097] In step S602 of some embodiments, the system compares the parsed appearance features and location information with the data in the database to determine whether the location information of the same item has changed, and filters out the item information with changed location information; the system generates corresponding request information based on the item information with changed location information, and displays the generated request information to the user through voice broadcast or a display device so that the user can confirm whether the item has moved.

[0098] In step S603 of some embodiments, the system detects the request information input by the user in real time, parses the request information to determine the item information for which the user determines that the location information has changed, and the system performs re-identification detection based on the item information determined by the user to determine its current location information, and updates the detected current location information to the database. Exemplarily, the system can also detect the items in the working area in real time. When it detects that the location of an item has changed, it requests confirmation from the user. After the user confirms the movement of the item through a voice command or manual operation, the system automatically detects the current location of the item and synchronizes it to all devices of the system.

[0099] Please refer to Figure 7 , a method for finding items based on interaction provided by an embodiment of the present invention may include but is not limited to steps S701 to S703:

[0100] Step S701, determine the target item according to the user instruction, query the database according to the target item to determine the location information of the target item;

[0101] Step S702, perform path planning according to the target location information, preset storage area information, and system map to determine the storage path;

[0102] Step S703, generate a driving instruction according to the storage path, and send the driving instruction to the storage robot so that the storage robot moves the target item to the preset storage area according to the driving instruction.

[0103] In step S701 of some embodiments, the system can also be connected to the storage robot, and the system drives the storage robot to perform storage management on the items in the working area according to the initialized environmental map; the system receives the user's voice instruction, parses the voice instruction to determine the user's intention; when the system determines through parsing that the user wants to store an item, such as "help me put the water cup back on the table", the system queries the database or detects the current environment to determine the location information of the "water cup".

[0104] In step S702 of some embodiments, the system determines the position of the item to be stored, determines the target storage position, performs path planning based on the established three-dimensional map, and drives the storage robot to perform the storage operation on the item according to the planned path; in this embodiment, the system generates a drive instruction according to the planned path, sends the drive instruction to the storage robot, and the storage robot can move the target item to the specified storage area according to the instruction.

[0105] In step S703 of some embodiments, the storage robot can be equipped with sensors to continuously sense the surrounding environment and locate its own position information; after receiving the drive instruction sent by the system, based on the map data synchronized by the system, it performs the storage operation on the target item according to the path of the instruction; during the movement and storage process, the robot continuously senses its own environment through the sensors, automatically avoids obstacles on the path, and maintains its own stability during the movement, improving the user experience.

[0106] Next, in combination with specific application examples, the solutions of the embodiments of the present invention will be introduced and described in detail:

[0107] Please refer to Figure 8 , Figure 8It is a structural block diagram of an intelligent storage robot system to which an item search method based on interaction provided by an embodiment of the present invention is applied; this intelligent storage robot is applied in a home environment and includes a robot hardware configuration, a large model interaction module, a data storage and management module, a multimodal fusion module, and an environment perception and positioning module; among them, the robot hardware configuration is the basis of this intelligent storage robot system and includes a camera, a microphone, a processor, a voice output module, a storage module, a communication module, and various sensors, such as a lidar, an IMU module, etc.; among them, the camera and the microphone are used to collect visual information and voice information in the home environment in real time, for item recognition and understanding of user instructions, and to give a response feedback to the user through the voice output module; various sensors are used to sense changes in the environment around the robot, for precise positioning to improve the efficiency of item storage; the large model interaction module integrates large models of natural language processing and computer vision, processes the collected voice information through the natural language processing large model, understands complex voice instructions, accurately analyzes the user's intention, and converts the processing result into natural language to interact with the user; the computer vision large model analyzes the collected visual information, extracts the appearance features of items, so as to classify and label the items; the data storage and management module is used to store and manage data related to items in the home environment and has functions of automatic update and data synchronization. When the position or state of an item changes, the data is updated in real time, and all related devices are synchronously updated; the multimodal fusion module combines visual, voice, and sensor data to comprehensively identify and manage items in the home environment, and improves the recognition accuracy of items by cross-verifying data from different information sources; the environment perception and positioning module integrates various sensors to accurately position the surrounding environment and items in real time, detect changes in the relative position of items, and accurately position the robot's position in the home environment; scans the home environment in real time to construct an indoor map, so that the robot can identify the furniture layout and path planning in the home environment, and adjusts its own posture in real time through sensors to maintain stability during movement and improve efficiency.

[0108] Please refer to Figure 9 , Figure 9It is a schematic diagram of the step - by - step process for an intelligent storage robot system to manage the storage of items; before the intelligent storage robot manages the storage of items, it prompts the user to initialize the system. The user issues commands through voice instructions or the touch - display module, and the system is initialized. The robot hardware configuration and environmental perception and positioning module of the intelligent storage robot scan the home environment, establish a three - dimensional map of the home environment, and delimit the working area and boundary information in the constructed three - dimensional map according to the voice instructions issued by the user; then, the user places the items to be stored and managed in the designated area, and the robot actively detects the appearance of the items through sensors and prepares to record the operation; the user briefly describes the items to be recorded through voice and issues a recording command; the robot collects the image data of the items in the designated area through the camera and collects the user's voice data through the microphone; the robot processes the image data and voice data through the large - model interaction module, extracts the appearance feature information of the items from the image data, extracts the key information of the items from the voice data, such as name, type, etc., and the robot detects the accurate position information of the items through the sensors in the environmental perception and positioning module, generates index information from the extracted feature information, parsed key information, and position information, and stores it in the database; in actual daily use, the user asks the intelligent storage robot and issues a query voice command; the robot receives the voice command through the microphone, converts the voice command into text information through the voice recognition algorithm in the large - model interaction module, and analyzes the text information to determine the user's intention and the description information of the relevant items; then, the robot retrieves in the database according to the analyzed item description information and queries the target item information that matches the item description information; then, the robot inputs the queried target item information into the large - model interaction module for processing. First, it uses the natural language generation algorithm to convert the target item information into an easy - to - understand natural language text, and then converts the natural language text into a voice signal through the voice synthesis algorithm; the robot, through the microphone it is equipped with, gives feedback to the user according to the obtained voice information, informing the location information of the item; at the same time, the robot displays the natural language text on the display screen carried by the robot, marks the location of the item in the constructed three - dimensional map, and displays it through the display screen; or sends the natural language text to the user's mobile device, such as a mobile phone, tablet, etc.; the robot continuously detects the location information of the items in the home environment. When it detects that the location of the item has changed, it sends a confirmation request to the user to determine whether the item has moved; after receiving the confirmation information fed back by the user, the robot detects the current location of the item through the sensors of the environmental perception and positioning module, updates the relevant information in the database, and synchronizes the updated database data to the relevant devices.

[0109] Implementing the embodiments of the present invention includes the following beneficial effects: The embodiments of the present invention provide an interactive-based item search method, system, electronic device, and storage medium. This solution parses the user's voice search instruction to determine the name of the item the user needs to search for; queries the database based on the determined item name to determine the relevant information of the item the user needs to search for, including the location information of the item; then, processes the retrieved item-related information and a preset feedback template to determine the feedback information; responds to the user based on the determined feedback information to inform the user of the relevant information of the item to be searched for; by parsing the user's voice instruction to extract the key information of the item to be searched for, determining the location information of the item based on the key information and the database and providing a feedback response, the user can obtain the location information of the item by voice query or issuing an instruction, which is simple to operate and improves the user experience; searching for the item based on the information in the feedback response can shorten the time for the user to search for the item and improve the item search efficiency; the system regularly detects the items in the environment to determine their appearance features or location information, automatically confirms with the user when the appearance features or location information change, and automatically updates the corresponding data to ensure the timeliness and accuracy of the data; at the same time, data recording of the items is performed based on computer vision without the need to process the items, which is convenient to use and improves the user experience.

[0110] As Figure 10 shown, the embodiments of the present invention also provide an interactive-based item search system that can implement the above-mentioned interactive-based item search method. The system includes:

[0111] A first module for performing speech recognition on the received voice query instruction to determine the query content information; wherein, the query content information includes the item name and the corresponding description information;

[0112] A second module for querying the database based on the query content information to determine the query result; wherein, the query result includes the item information and the corresponding location information;

[0113] A third module for processing the query result and a preset feedback template to determine the response information; and responding based on the response information.

[0114] It can be seen that the content in the above method embodiments is applicable to the system embodiments of the present invention. The functions specifically implemented by the system embodiments of the present invention are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0115] An embodiment of the present application further provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the above-described method for finding an item based on interaction is implemented. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0116] It can be understood that the content in the above method embodiments is applicable to the device embodiments of the present application. The functions specifically implemented by the device embodiments of the present application are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0117] Please refer to Figure 11 , Figure 11 which schematically shows the hardware structure of an electronic device according to another embodiment. The electronic device includes:

[0118] A processor 1101, which can be implemented by using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present application;

[0119] A memory 1102, which can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 1102 can store an operating system and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1102, and the processor 1101 is used to call and execute a method for finding an item based on interaction according to an embodiment of the present application;

[0120] An input / output interface 1103, which is used to implement information input and output;

[0121] A communication interface 1104, which is used to implement communication and interaction between the present device and other devices, and can implement communication through a wired method (such as USB, network cable, etc.) or through a wireless method (such as a mobile network, WIFI, Bluetooth, etc.);

[0122] A bus 1105, which transmits information between various components of the device (such as the processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104);

[0123] Among them, the processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104 are communicatively connected to each other inside the device through the bus 1105.

[0124] Among them, the memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. The memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory optionally includes a remote memory that is remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0125] In addition, an embodiment of the present application also discloses a computer program product or a computer program, and the computer program product or the computer program is stored in a computer-readable storage medium. The processor of the computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned method. Similarly, the content in the above method embodiments is applicable to this storage medium embodiment, and the functions specifically implemented by this storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0126] An embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned method for finding an item based on interaction.

[0127] It can be understood that the content in the above method embodiments is applicable to this storage medium embodiment, and the functions specifically implemented by this storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0128] It can be understood that all or some of the steps and systems disclosed in the above methods can be implemented as software, firmware, hardware, and their appropriate combinations. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or can be implemented as hardware, or can be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include a computer storage medium (or non-transitory medium) and a communication medium (or transitory medium). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, a communication medium typically contains computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and can include any information delivery medium.

[0129] An item search method based on interaction, an item search system based on interaction, an electronic device, and a storage medium provided by an embodiment of the present invention parse a voice search instruction of a user to determine the name of the item that the user needs to search for; query in a database according to the determined item name to determine relevant information of the item that the user needs to search for, including the location information of the item; then, process according to the obtained item relevant information and a preset feedback template to determine feedback information; respond to the user according to the determined feedback information to inform the user of the relevant information of the item to be searched for; by parsing the user's voice instruction to extract the key information of the searched item, determine the location information of the item according to the key information and the database and perform a feedback response, the user can obtain the location information of the item by voice query or issuing an instruction, the operation is simple, and the user experience is improved; searching for the item according to the information of the feedback response can shorten the time for the user to search for the item and improve the item search efficiency; the system regularly detects the items in the environment to determine their appearance features or location information, and when the appearance features or location information change, automatically confirm with the user and automatically update the corresponding data to ensure the timeliness and accuracy of the data.

[0130] The above is a specific description of the preferred embodiment of the present invention. However, the present invention is not limited to the described embodiment. Those skilled in the art can make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included within the scope defined by the claims of this application.

Claims

1. An interactive object search method, characterized in that: The method comprises: Performing voice recognition on the received voice query instruction to determine the query content information; wherein the query content information includes the item name and the corresponding description information; Querying a database according to the query content information to determine a query result; wherein the query result includes item information and corresponding location information; Processing is performed according to the query result and a preset feedback template to determine response information; and responding is performed according to the response information.

2. The method according to claim 1, characterized in that The step of performing voice recognition on the received voice query instruction to determine the query content information specifically includes: Performing frame processing on the voice query command to determine a plurality of groups of frame signals, performing feature extraction on the plurality of groups of frame signals to determine a time series feature set; Extracting time series features from the time series feature set to determine a feature information set; mapping the feature information set to determine a label probability set; wherein the feature information set includes time series dependency and context relationship information; The tag probability set is decoded to determine a query text, and the query text is used as the query content information.

3. The method according to claim 1, characterized in that: The processing according to the query result and the preset feedback template to determine the response information specifically includes: According to the query result, the feedback text information is matched with the preset feedback template to determine; Extracting features from the feedback text information to determine text feature information; modeling based on the text feature information to determine a speech feature data set; Perform speech synthesis according to the speech feature data set to determine speech waveform data; and use the speech waveform data as the response information.

4. The method according to claim 1, characterized in that The method further comprises: Obtaining an initialization instruction, parsing the initialization instruction, and determining initialization information; wherein the initialization information includes working area and boundary information; Perform environmental scanning according to the initialization instruction to determine environmental data; perform map modeling according to the environmental data to determine three-dimensional map data; configure the three-dimensional map data according to the initialization information to determine the system map.

5. The method according to claim 1, characterized in that The method further comprises: Parsing the received recording instruction to determine the item information; wherein the item information includes the item name and item location information; Collect data according to the recording instruction to determine an object image set; extract features from the object image set to determine an object feature set; Generate index information according to the item information and the item feature set, and save the index information to the database.

6. The method according to claim 1, characterized in that The method further comprises: Scan the current environment according to a preset period to determine a current environment data set; parse the current environment data set to determine current item information data; Compare the current item information data with the database to determine abnormal item information; generate request information according to the abnormal item information; wherein the abnormal item information includes item information with changed location information, and the request information is used by the user to confirm the change in location information; Acquire feedback information, and update the database according to the feedback information and the abnormal item information; wherein the feedback information includes item information whose location information has changed as confirmed by the user.

7. The method according to claim 1, characterized in that The method further comprises: Determine the target object according to the user instruction, query the database according to the target object, and determine the location information of the target object; Perform path planning according to the target location information, the preset storage area information and the system map to determine the storage path; A driving instruction is generated according to the storage path, and the driving instruction is sent to the storage robot, so that the storage robot moves the target object to a preset storage area according to the driving instruction.

8. An interactive item search system, characterized in that: include: The first module is used to perform voice recognition on the received voice query instruction to determine the query content information; wherein the query content information includes the name of the item and the corresponding description information; The second module is used to query the database according to the query content information to determine the query result; wherein the query result includes item information and corresponding location information; The third module is used to process according to the query result and the preset feedback template to determine the response information; and respond according to the response information.

9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 7 when executed by the processor.

Images