Article search method, article search device, and storage medium

Abstract

The present disclosure relates to an article searching method, an article searching device and a storage medium. The article searching method is applied to an article searching device. The article searching device has one or more working modes, and different working modes correspond to different article searching operations. An article searching instruction is received from a user. The article searching instruction is analyzed to determine an article searching operation expected by the user. A working mode capable of performing the article searching operation is determined, and the article searching operation is performed in the working mode. By providing different working modes, the flexibility of the article searching device in performing the article searching operation is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to an object search method, an object search device, and a storage medium. Background Art

[0002] In our daily lives, we often need to find objects. With the advancement of technology, in order to save users time searching for objects, related technologies have installed a transmitter on the object. When the user needs to find the object, a signal is transmitted to the object via a handheld terminal. After receiving the transmitted signal, the object responds via the transmitter installed on the object. This response can be achieved by emitting an audible or visual alarm signal, which the handheld terminal uses to locate the object. The user then finds the object based on the positioning result of the handheld terminal. However, the object search process in related technologies not only requires user participation, but also has a single search mode and low flexibility. Summary of the Invention

[0003] To overcome the problems existing in the related art, the present disclosure provides an object finding method, an object finding device and a storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an object finding method, which is applied to an object finding device, wherein the object finding device has one or more operating modes, wherein different operating modes correspond to performing different object finding operations. The object finding method includes:

[0005] Receive an item search instruction issued by a user; parse the item search instruction to determine the item search operation expected by the user; determine a working mode that can perform the item search operation, and perform the item search operation in the working mode.

[0006] In one embodiment, the object-finding device has an object-finding mode, and the object-finding operation in the object-finding mode includes a single searching operation; performing the object-finding operation in the working mode includes: determining the target object corresponding to the object-finding operation, and querying the location of the target object; if the location of the target object is queried, searching for the target object according to the queried location; if the location of the target object is not queried, collecting environmental images within a specified searching range, and searching for the target object based on the visual features of the objects in the collected environmental images and the object features of the target object.

[0007] In one embodiment, capturing an environmental image within a specified search range, and searching for the target object based on visual features of the object in the captured environmental image and object features of the target object, includes:

[0008] Traversing each room location in the stored room map, and determining a search path for performing the single search operation based on each room location; collecting environmental images along the search path, and extracting visual features of objects included in the environmental images; matching the visual features of the object with the object features of the target item; if the visual features of the object match the object features of the target item, determining that the item has been found; if the visual features of the object do not match the object features of the target item, continuing to collect environmental images along the search path until the search path is traversed or the item is found.

[0009] In one embodiment, the object-finding device has a picking-up mode, and the object-finding operation in the picking-up mode includes a searching operation and a picking-up operation; executing the object-finding operation in the working mode includes: parsing the object-finding instruction, determining the target object specified by the user to be picked up, and determining the location of the target object; planning a picking-up path to reach the location of the target object, and reaching the location of the target object along the picking-up path; collecting an environmental image at the location of the target object, and extracting the object visual features included in the environmental image; matching the object visual features in the collected environmental image with the object features of the target object; and determining the object-finding result of executing the object-finding operation based on the matching result, and the object-finding result includes finding the target object and not finding the target object.

[0010] In one embodiment, the object-finding device has a homing mode, and the object-finding operation in the homing mode includes an object-homing operation; executing the object-finding operation in the working mode includes: parsing the object-finding instruction, the object-finding instruction includes a homing position indicating that the target object is to be returned; placing the target object in the homing position.

[0011] In one embodiment, the object search method further includes: parsing the object search instruction; if the user's instruction to return the object to the location is not parsed, outputting the placement location of the target object when the target object is found.

[0012] In one embodiment, the object search method further includes: if the target object is not found, outputting a prompt message, wherein the prompt message is used to prompt that the target object is not found.

[0013] According to a second aspect of an embodiment of the present disclosure, there is provided an object finding device, which is applied to an object finding device. The object finding device has one or more operating modes, wherein different operating modes correspond to performing different object finding operations. The object finding device includes:

[0014] A receiving unit is used to receive an item search instruction issued by a user; a parsing unit is used to parse the item search instruction and determine the item search operation expected by the user; and a determining unit is used to determine a working mode that can perform the item search operation and perform the item search operation in the working mode.

[0015] In one embodiment, the object-finding device has an object-finding mode, and the object-finding operation in the object-finding mode includes a single searching operation; the determination unit is used to: determine the target object corresponding to the object-finding operation, and query the location of the target object; if the location of the target object is queried, search for the target object according to the queried location; if the location of the target object is not queried, collect environmental images within a specified searching range, and search for the target object based on the visual features of the objects in the collected environmental images and the object features of the target object.

[0016] In one embodiment, the determination unit is used to: traverse each room location in the stored room map, and determine a search path for performing the single search operation based on each room location; capture environmental images along the search path, and extract visual features of the objects included in the environmental images; match the visual features of the objects with the object features of the target item; if the visual features of the object match the object features of the target item, it is determined that the item is found; if the visual features of the object do not match the object features of the target item, continue to capture environmental images along the search path until the search path is traversed or the item is found.

[0017] In one embodiment, the object-finding device has a picking-up mode, and the object-finding operation in the picking-up mode includes a searching operation and a picking-up operation; the determination unit is used to: parse the object-finding instruction, determine the target object specified by the user to be picked up, and determine the location of the target object; plan a picking-up path to the location of the target object, and reach the location of the target object along the picking-up path; collect an environmental image at the location of the target object, and extract the object visual features included in the environmental image; match the object visual features in the collected environmental image with the object features of the target object; and determine the object-finding result of executing the object-finding operation based on the matching result, and the object-finding result includes finding the target object and not finding the target object.

[0018] In one embodiment, the object-finding device has a homing mode, and the object-finding operation in the homing mode includes an object-homing operation; the determination unit is used to: parse the object-finding instruction, and the object-finding instruction includes a homing position indicating that the target object is to be returned; and place the target object at the homing position.

[0019] In one embodiment, the parsing unit is configured to: parse the item search instruction; if the user's instruction to return the item to the location indicated by the user is not parsed, output the placement location of the target item when the target item is found.

[0020] In one embodiment, the determining unit is configured to: if the target object is not found, output a prompt message, wherein the prompt message is used to indicate that the target object is not found.

[0021] According to a third aspect of an embodiment of the present disclosure, there is provided an object finding device, comprising:

[0022] A processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the item search method described in any one of the embodiments of the first aspect.

[0023] According to the fourth aspect of the embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by the processor of the object-finding device, the object-finding device can execute the object-finding method described in any one of the embodiments of the first aspect.

[0024] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects: receiving and parsing a user's item search instruction, determining the item search operation desired by the user, determining an operating mode capable of performing the item search operation among different operating modes, and performing the item search operation in that operating mode. Thus, by providing different operating modes, the flexibility of the item search device in performing item search operations is increased.

[0025] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0027] Figure 1 The figure is a flowchart of a method for finding an item according to an exemplary embodiment of the present disclosure.

[0028] Figure 2 FIG. 4 is a flowchart of performing a single searching operation in a search mode according to an exemplary embodiment of the present disclosure.

[0029] Figure 3 The figure is a flowchart showing a method of searching for a target object by using an environment image according to an exemplary embodiment of the present disclosure.

[0030] Figure 4 FIG. 4 is a flowchart showing an example of performing an item search operation in a pickup mode according to an exemplary embodiment of the present disclosure.

[0031] Figure 5 The figure is a flowchart showing an example of performing an item search operation in a homing mode according to an exemplary embodiment of the present disclosure.

[0032] Figure 6 The figure is a block diagram of an item search device according to an exemplary embodiment of the present disclosure.

[0033] Figure 7 The figure is a block diagram of a device for finding an item according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0035] People often place items they use away from their hands, only to forget where they left them the next time they need them. To address this issue, related art proposes attaching a transmitter to an item. When a user needs to find the item, they transmit a signal to the item using a handheld terminal. Upon receiving the signal, the item responds via the transmitter. This response may include emitting an audible or visual alarm signal or a signal describing its current location. The handheld terminal locates the item based on the response signal transmitted by the item. The user then finds the item based on the handheld terminal's location results. However, few of today's everyday items have transmitters. Therefore, implementing related art requires installing transmitters on the item to be found. Installing transmitters on the item to be found requires significant effort, high investment costs, and ongoing maintenance. Furthermore, using the methods provided in related art requires user involvement when searching for an item. In some cases, users may know where an item is located but are unable to retrieve it due to mobility issues or lack of time. Related art cannot implement the operation of picking up an item, and suffers from a limited search mode and low flexibility.

[0036] In view of this, an embodiment of the present disclosure provides an object search method for searching for objects within a specified search range. A search device that applies the object search method should have one or more working modes. The object search device can receive and parse the object search instructions issued by the user through the object search method provided by the present disclosure, and determine the object search operation desired by the user. In different working modes, a working mode that can perform the object search operation is determined. The object search operation desired by the user is performed in the working mode to find the target object specified by the user. In this way, a variety of working modes are provided for users to choose from, which increases the flexibility of the object search device.

[0037] It should be understood that the object-finding device referred to in this disclosure can be any existing robot with one or more operating modes. The robot is capable of performing various basic actions based on motion instructions generated by a processor. These basic actions include moving forward, turning, and retrieving objects. In other words, any robot / machine device / intelligent robot that utilizes the object-finding method provided in this disclosure should be protected by this disclosure.

[0038] It should be noted that when the object-finding device in the present disclosure executes the object-finding method, it can use the simultaneous localization and mapping (SLAM) algorithm in advance to build a map within the specified search range, which is used for path planning and navigation during the object-finding operation using the constructed map. When building the map, the object-finding device in the present disclosure captures the environmental image in the field of view of the object-finding device based on the camera device. Objects in the environmental image are detected and identified (object detection), and the spatial relationship (spatial relation correlation) between different objects in the environmental image is inferred. The spatial relationship results between different objects obtained by reasoning are stored. In this embodiment, the detection and identification of objects in the environmental image and the reasoning of the spatial relationship between different objects can be performed in advance.

[0039] The following embodiments will illustrate an object search method provided by the present disclosure in conjunction with the accompanying drawings.

[0040] Figure 1 FIG. 1 is a flow chart of a method for finding an item according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the object finding method is applied to an object finding device, which has one or more working modes, wherein different working modes correspond to performing different object finding operations. The object finding method provided by the present disclosure includes the following steps.

[0041] In step S11, an item search instruction issued by a user is received.

[0042] The item finding instruction issued by the user can be a text message input into the item finding device, or it can be a voice instruction issued by the user. The item finding device receives the voice instruction issued by the user through a voice receiving device, or the text information input through the control terminal connected to the item finding device. The user instructs the item finding robot to perform the item finding operation desired by the user through the item finding instruction issued. For example, when the user issues the item finding instruction "Xiao Ai, help me find the blue socks" in the form of language, the item finding device receives the voice instruction issued by the user through the voice receiving device. For another example, when the user inputs "help me get xxx next to xx in room xx" on the control terminal connected to the item finding device, the control device of the item finding device receives the text information transmitted by the control terminal. On a item finding device with an input interface, the user can directly input the item finding instruction.

[0043] In step S12, the item search instruction is parsed to determine the item search operation expected by the user.

[0044] Utilize the natural language processing model to parse the item search instruction. The natural language processing model determines the user's semantics by performing text classification, keyword extraction, syntactic analysis and other natural language processing (NLP) technologies on the item search instruction. Determine the item search operation expected by the user based on the user's semantics. The item search operation in the present disclosure includes searching for items, searching for items and placing the found items in a specified location, searching for items and informing the location information of the found items and picking up items, etc. For example, the natural language understanding model is used to parse the item search instruction "Xiao Ai, help me find blue socks". Based on the keywords "search", "blue" and "socks", it is determined that the item search operation expected by the user is to search for items and search for "socks" with the "blue" feature. For example, the natural language processing model can adopt a self-encoding language model (Bidirectional Encoder Representations from Transformers, BERT). It should be understood that the natural language processing model is pre-trained using training data.

[0045] In step S13, a working mode in which the object search operation can be performed is determined, and the object search operation is performed in the working mode.

[0046] In the disclosed embodiment, a correspondence between operating modes and item search operations is predefined. After the item search command is parsed to obtain the item search operation desired by the user, the operating mode corresponding to the desired item search operation is determined from the correspondence. The item search operation is then executed in the determined operating mode.

[0047] Through the object finding method provided by the present disclosure, a finder device can automatically perform the object finding operation without user intervention. Compared to related technologies, the found object does not need to be equipped with a transmitter, and can still help users find the desired item quickly and effectively. This saves users time in searching for the desired item, improves the user experience, and increases the efficiency of the finder device.

[0048] In different working modes, the object-finding device can perform different object-finding operations, thereby improving the flexibility of the object-finding device. This embodiment will respectively illustrate the object-finding operations performed by the object-finding device in different working modes.

[0049] In one embodiment, the object-finding device has an object-finding mode, and the object-finding operation in the object-finding mode includes a single searching operation. Figure 2 FIG. 1 is a flow chart showing a single search operation in a search mode according to an exemplary embodiment of the present disclosure. Figure 2 As shown, performing a single search operation in the object-finding mode includes the following steps.

[0050] In step S21 , a target object corresponding to the object search operation is determined, and the location of the target object is inquired.

[0051] Parse the item search instruction issued by the user and determine the target item to be searched from the parsed result. For example, in the above-mentioned item search instruction "Xiao Ai, help me find blue socks", the target item the user is looking for is socks. The location of the target item is stored in the map constructed by the item search device. When the item search device determines the target item, it can search for the location of the target item in the stored map. In one example, the item search device can query the location of the target item in the stored map, or it can determine the search range for the target item specified by the user in the item search instruction. For example, the user specifies to search for the target item socks in the bedroom. Execute the item search operation based on the search result.

[0052] In step S22, if the location of the target object is found, the target object is searched according to the found location.

[0053] Since the target item may be frequently used and placed by the user, the locations of multiple target items may be queried. If the locations of multiple target items are queried, the locations of the multiple target items are sorted according to preset conditions, and the queried locations are reached in sequence to search for the target item. There will be two search results when searching for the target item at the queried location, and the two search results include finding the target item or not finding the target item. It can be understood that, also due to the use of the user, the object-finding device finds that the target item is not at the location after arriving at the location. If the search result is that the target item is not found, the environment image is collected within the specified search range, and the target item is searched based on the visual features of the items in the collected environment image and the object features of the target item.

[0054] In step S23 , if the location of the target object is not found, an environmental image is collected within the specified search range, and the target object is searched based on the visual features of the objects in the collected environmental image and the object features of the target object.

[0055] In the embodiment of the present disclosure, Figure 3 Description: Use the collected environment images to search for target objects within the specified search range. Figure 3 FIG. 1 is a flow chart showing how to use an environment image to find a target object according to an exemplary embodiment of the present disclosure. Figure 3 As shown, collecting environmental images within a specified search range, and searching for a target object based on the visual features of the objects in the collected environmental images and the object features of the target object, include the following steps.

[0056] In step S231 , each room position in the stored room map is traversed, and a search path for performing a single search operation is determined based on each room position.

[0057] In step S232 , the environment image is collected along the search path, and the visual features of the objects included in the environment image are extracted.

[0058] Extract visual features of all candidate objects in the environment image.

[0059] In step S233 , the object visual features are matched with the object features of the target item.

[0060] In an embodiment of the present disclosure, a multimodal model is used to determine whether the target object is included in the environmental image. The visual features of the candidate object and the object features of the target object are input into the multimodal object search discrimination model. The multimodal model outputs whether the candidate object is the target object expected by the user. In the above-mentioned object search instruction "Xiao Ai, help me find blue socks", "blue" and "socks" are both text features of the target object. The environmental image captured by the object search device during the object search process includes the visual features of the object. The multimodal model is used to discriminate the text features of the target object and the visual features of the candidate object to obtain a discrimination result. By combining the text features and visual features of the object, the probability of successfully finding the object is increased. For example, the multimodal model can be implemented using a visual autoencoder language model (Visual BERT). The multimodal model can be pre-trained using relevant technologies.

[0061] In step S234 , if the visual features of the object match the object features of the target item, it is determined that the item has been found.

[0062] In step S235 , if the visual features of the object do not match the object features of the target item, the environment image is continuously collected along the search path until the search path is traversed or the item is found.

[0063] In one embodiment, the object-finding device has a picking-up mode, and the object-finding operation in the picking-up mode includes a searching operation and a picking-up operation. Figure 4 Describes the process of performing an item search operation in the Pickup Mode. Figure 4 FIG. 1 is a flowchart illustrating an example of performing an item search operation in a pickup mode according to an exemplary embodiment of the present disclosure. Figure 4 As shown, performing an item search operation in the picking-up mode includes the following steps.

[0064] In step S31, the object search instruction is parsed to determine the target object specified by the user to be picked up and the location of the target object is determined.

[0065] The above item search instruction "Help me get xxx next to xx in room xx" is parsed to obtain the target item to be picked up by the user and the location of the target item specified by the user. The item search operation to be performed by the search device includes a search operation and a pick-up operation. This can be understood as the need to first find the target item in the pick-up mode and then perform the pick-up operation. For example, if the user does not specify the location of the target item, the search device can query the constructed map to determine the location of the target object.

[0066] In step S32, a picking-up path to the location of the target object is planned, and the target object is located along the picking-up path.

[0067] In step S33, an environmental image is collected at the location of the target object, and visual features of the object included in the environmental image are extracted.

[0068] In step S34, the visual features of the objects in the collected environment image are matched with the object features of the target object.

[0069] In step S35 , the item search result of performing the item search operation is determined based on the matching result. The item search result includes whether the target item is found or not found.

[0070] In the picking mode, if the target item specified by the user is not at the location where the user specifies the target item, or the target item is not at the location where the target object is queried by the finding device, the item finding result may be that the target item is not found.

[0071] In one embodiment, the object-finding device has a homing mode, and in the homing mode, the object-finding operation includes the object-homing operation. Figure 5 Describes the process of performing an item search operation in homing mode. Figure 5 FIG. 1 is a flowchart illustrating an example of performing an item search operation in a homing mode according to an exemplary embodiment of the present disclosure. Figure 5 As shown, performing an item search operation in the homing mode includes the following steps.

[0072] In step S41, the item search instruction is parsed.

[0073] In the embodiment of the present disclosure, the object search instruction includes a homing position indicating that the target object should be returned to its original position.

[0074] In real life, there are often users with limited mobility. To make their lives more convenient, a finder can send an item-finding instruction to the finder, including instructions for returning the target item. By parsing the item-finding instruction, the user can determine the location where the item is to be returned.

[0075] In this embodiment, the homing mode can be used in conjunction with the object picking mode and the object finding mode. For example, the object finding instruction includes the target object to be found and also includes the homing position for returning the target object. Or the object finding instruction includes the target object to be picked up specified by the user and the homing position for returning the target object to be picked up. The homing mode can also wait for the user to issue the object finding instruction again after executing the object finding operation corresponding to the object finding mode and finding the target object. The object finding instruction issued by the user again includes the homing position for returning the target object. The object finding instruction issued by the user again is parsed, and the object finding operation is performed in the homing mode. It should be understood that the present disclosure does not limit the combination of different modes.

[0076] In step S42 , the target object is placed in a home position.

[0077] For example, the object-finding device picks up a target object and plans a homing path based on the homing location. The target object is placed at the homing location along the homing path. After the target object is placed at the homing location, the user is notified to place the target object at the homing location.

[0078] In one embodiment, the object search method of the present disclosure further includes: parsing the object search instruction; if the location where the user instructs to return the object is not parsed, outputting the placement location of the target object when the target object is found.

[0079] This disclosure utilizes a visual language navigation model to output the current location of a target object. The visual language navigation model generates a natural language navigation description based on the spatial location of the target object and the current location of the finder (or user). For example, "The key is under the kitchen chair." The visual language navigation model can be pre-trained using training data.

[0080] In addition to outputting the location of the target object through the visual language navigation model, the present invention can also send the location of the target object to a terminal connected to the object-finding device to facilitate the user to query the search results.

[0081] In one embodiment, the object search method provided by the present disclosure further includes: if the target object is not found, outputting a prompt message. The prompt message is used to indicate that the target object has not been found.

[0082] The object search method provided by the present disclosure uses the camera device and navigation system of the object search device to enable the object search device to complete the task of finding the target object by itself, without the need for the user to hold the terminal and search for the target object based on the signal fed back by the terminal. In addition, the present disclosure relies on the camera device, navigation system, and natural language related algorithms to accurately locate the object being searched, or generate an accurate location description (such as "xxx is on the left side of the kitchen sink") so that the user can access it at their convenience. This saves the user time in finding items and improves the user experience.

[0083] Based on the same concept, an embodiment of the present disclosure also provides an object finding device.

[0084] It is understandable that the object search device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0085] Figure 6 This is a block diagram of an item search device according to an exemplary embodiment of the present disclosure. Figure 6 The object finding device 100 is applied to an object finding device having one or more working modes, wherein different working modes correspond to performing different object finding operations. The object finding device includes a receiving unit 101, a parsing unit 102 and a determining unit 103.

[0086] The receiving unit 101 is used to receive an item search instruction issued by a user; the parsing unit 102 is used to parse the item search instruction and determine the item search operation expected by the user; the determining unit 103 is used to determine a working mode that can perform the item search operation and perform the item search operation in the working mode.

[0087] In one embodiment, the object-finding device has an object-finding mode, and the object-finding operation in the object-finding mode includes a single searching operation; the determination unit 103 is used to: determine the target object corresponding to the object-finding operation, and query the location of the target object; if the location of the target object is queried, the target object is searched according to the queried location; if the location of the target object is not queried, the environmental image is collected within the specified searching range, and the target object is searched based on the visual features of the object in the collected environmental image and the object features of the target object.

[0088] In one embodiment, the determination unit 103 is used to: traverse each room location in the stored room map, and determine a search path for performing a single search operation based on each room location; capture environmental images along the search path, and extract visual features of the object included in the environmental image; match the visual features of the object with the object features of the target item; if the visual features of the object match the object features of the target item, it is determined that the object is found; if the visual features of the object do not match the object features of the target item, continue to capture environmental images along the search path until the search path is traversed or the object is found.

[0089] In one embodiment, the object-finding device has a picking-up mode, and the object-finding operation in the picking-up mode includes a searching operation and a picking-up operation; the determination unit 103 is used to: parse the object-finding instruction, determine the target object specified by the user to be picked up, and determine the location of the target object; plan a picking-up path to the location of the target object, and reach the location of the target object along the picking-up path; collect an environmental image at the location of the target object, and extract the object visual features included in the environmental image; match the object visual features in the collected environmental image with the object features of the target object; and determine the object-finding result of executing the object-finding operation based on the matching result, and the object-finding result includes finding the target object and not finding the target object.

[0090] In one embodiment, the object-finding device has a homing mode, and the object-finding operation in the homing mode includes an object-homing operation; the determination unit 103 is used to: parse the object-finding instruction, the object-finding instruction includes a homing position indicating that the target object is to be returned; and place the target object at the homing position.

[0091] In one embodiment, the parsing unit 102 is configured to: parse the object search instruction; if the user's instruction to return the object to the location is not parsed, output the placement location of the target object when the target object is found.

[0092] In one embodiment, the determining unit 103 is configured to: if the target object is not found, output a prompt message, where the prompt message is used to indicate that the target object is not found.

[0093] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0094] Figure 7 2 is a block diagram illustrating an apparatus for finding an item according to an exemplary embodiment of the present disclosure. For example, apparatus 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0095] Reference Figure 7 , apparatus 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0096] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0097] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0098] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 200.

[0099] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0100] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC) that is configured to receive external audio signals when the device 200 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 204 or transmitted via the communication component 216. In some embodiments, the audio component 210 further includes a speaker for outputting audio signals.

[0101] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0102] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200. The sensor assembly 214 can also detect changes in the position of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200, and temperature changes of the device 200. The sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 214 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0103] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0104] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method.

[0105] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as the memory 204 including instructions, which can be executed by the processor 220 of the apparatus 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0106] It is understood that in this disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of related objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0107] It will be further understood that the terms "first," "second," and the like are used to describe various types of information, but such information should not be limited to these terms. These terms are used solely to distinguish information of the same type from one another and do not indicate a particular order or level of importance. In fact, the terms "first," "second," and the like are fully interchangeable. For example, first information could be referred to as second information, and similarly, second information could be referred to as first information without departing from the scope of this disclosure.

[0108] It is further understood that, unless otherwise specified, “connection” includes a direct connection where there are no other components between the two elements, and also includes an indirect connection where there are other elements between the two elements.

[0109] It is further understood that although operations are described in a particular order in the drawings in the embodiments of the present disclosure, this should not be construed as requiring that the operations be performed in the particular order shown or in a serial order, or that all of the operations shown be performed to obtain the desired results. In certain circumstances, multitasking and parallel processing may be advantageous.

[0110] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0111] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.

Claims

1. A method for finding an object, characterized in that: Applied to a search device, the search device has multiple working modes, wherein different working modes correspond to performing different object search operations, and the object search method includes: Receive item search instructions from users; Parsing the item search instruction to determine the item search operation expected by the user; determining a working mode capable of performing the item search operation, and performing the item search operation in the working mode; The working mode includes a search mode and a pick-up mode. The search operation in the search mode includes a single search operation, and the search operation in the pick-up mode includes a search operation and a pick-up operation. The performing of the object search operation in the working mode includes: performing a single search operation in the object search mode; The single search operation in the object-finding mode includes: Determining a target object corresponding to the object search operation and querying a location of the target object; If the location of the target item is found, the target item is searched for according to the found location; If the location of the target object is not found, an environmental image is collected within a specified search range, and the target object is searched for based on the visual features of the object in the collected environmental image and the object features of the target object.

2. The object search method according to claim 1, characterized in that: Collecting an environmental image within a specified search range, and searching for the target object based on visual features of the object in the collected environmental image and object features of the target object, including: Traversing each room position in the stored room map, and determining a search path for performing the single search operation based on each room position; Acquiring an environmental image along the search path, and extracting visual features of objects included in the environmental image; matching the object visual features with the object features of the target item; If the visual features of the object match the object features of the target item, it is determined that the item has been found; If the visual features of the object do not match the object features of the target item, then continue to collect environmental images along the search path until the search path is traversed or the item is found.

3. The object search method according to claim 1, wherein: The object-finding device has an object-picking mode; Executing the item search operation in the working mode includes: Parsing the item search instruction, determining the target item specified by the user to be picked up, and determining the location of the target item; Planning a picking path to the location of the target item, and reaching the location of the target item along the picking path; Collecting an environmental image at the location of the target object and extracting visual features of the object included in the environmental image; Matching the visual features of the object in the collected environment image with the object features of the target object; Based on the matching results, an item search result of performing the item search operation is determined, where the item search result includes whether the target item is found or not found.

4. The object search method according to any one of claims 1 to 3, characterized in that: The object-finding device has a homing mode, and the object-finding operation in the homing mode includes an object-returning operation; Executing the item search operation in the working mode includes: parsing the object search instruction, wherein the object search instruction includes a homing position indicating where to return the target object; The target object is placed in the home position.

5. The object search method according to any one of claims 1 to 3, characterized in that: The item search method further includes: Parsing the item search instruction; If the user's instruction to return the item to the returning location is not resolved, the placement location of the target item is output when the target item is found.

6. The object search method according to any one of claims 1 to 3, characterized in that: The item search method further includes: If the target object is not found, a prompt message is output, where the prompt message is used to prompt that the target object is not found.

7. An object finding device, characterized in that: Applied to a search device, the search device has multiple working modes, wherein different working modes correspond to performing different object search operations, and the object search device includes: A receiving unit, configured to receive an item search instruction from a user; a parsing unit, configured to parse the item search instruction and determine the item search operation expected by the user; a determining unit, configured to determine a working mode capable of performing the item search operation, and perform the item search operation in the working mode; The working mode includes a search mode and a pick-up mode. The search operation in the search mode includes a single search operation, and the search operation in the pick-up mode includes a search operation and a pick-up operation. The determining unit performs a single search operation in the object-finding mode in the following manner: Determining a target object corresponding to the object search operation and querying a location of the target object; If the location of the target item is found, the target item is searched for according to the found location; If the location of the target object is not found, an environmental image is collected within a specified search range, and the target object is searched for based on the visual features of the object in the collected environmental image and the object features of the target object.

8. An object finding device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the object search method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium stores instructions. When the instructions in the storage medium are executed by a processor of the object-finding device, the object-finding device is enabled to execute the object-finding method according to any one of claims 1 to 6.

Images