Object searching method, self-moving device, computer storage medium and electronic device

Abstract

The present disclosure relates to the technical field of smart home. Provided are an object searching method, a self-moving device, a computer storage medium and an electronic device. The object searching method is applied to the self-moving device, and the self-moving device is provided with an image acquisition device. The method comprises: after an object searching instruction is received, by means of the image acquisition device, collecting a field-of-view image of the self-moving device during a movement process, the object searching instruction comprising feature description information of a target object to be searched; on the basis of the feature description information, detecting whether the field-of-view image contains the target object, so as to obtain an object detection result; and, on the basis of the object detection result, determining an object searching result for the target object. The object searching method in the present disclosure can overcome the limitation that only objects having signal transmitting capabilities can be searched in the prior art.

Description

Object search method, self-mobile device, computer storage medium, electronic device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese patent application number 202410024179.8, filed on January 5, 2024, entitled “Object Search Method, Self-Mobile Device, Computer Storage Medium, Electronic Device,” and the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of smart home technology, and in particular to an object search method, a mobile device, a computer storage medium, and an electronic device. Background Art

[0004] With the rapid development of smart home technology, mobile devices are bringing convenience to more and more families.

[0005] In the related art, some self-moving devices have an automatic return to charging pile function. However, the self-moving device can only find the charging pile based on the signal sent by the charging pile. Once the signal is not received, the above-mentioned automatic return to charging pile function cannot be realized.

[0006] In view of this, there is an urgent need to develop a new object search method in this field.

[0007] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of this disclosure. Summary of the Invention

[0008] The purpose of the present disclosure is to provide an object search method, a mobile device, a computer storage medium and an electronic device, thereby overcoming, at least to a certain extent, the technical problem of only being able to search for objects capable of emitting signals due to limitations of related technologies.

[0009] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0010] According to a first aspect of the present disclosure, there is provided an object search method, which is applied to a self-mobile device, wherein the self-mobile device is provided with an image acquisition device, and the method comprises:

[0011] After receiving the object search instruction, the image capture device captures the viewfinder image of the mobile device during movement; the object search instruction includes feature description information of the target object to be found;

[0012] The target object is detected in the viewfinder image according to the feature description information, an object detection result is obtained, and an object search result for the target object is determined based on the object detection result.

[0013] In an exemplary embodiment of the present disclosure, after receiving the object search instruction, capturing, by the image capture device, a viewfinder image of the self-mobile device during movement, includes:

[0014] After receiving the object search instruction, determining an initial sub-area to be searched from the current scene;

[0015] The image acquisition device is used to acquire an initial framing image of the self-moving device during its movement within the initial sub-area to be found.

[0016] In an exemplary embodiment of the present disclosure, determining an initial sub-area to be searched from the current scene includes:

[0017] Determining an initial position of the mobile device in the current scene;

[0018] A circular area with the initial position as the center and a preset distance as the radius is determined as the initial sub-area to be found.

[0019] In an exemplary embodiment of the present disclosure, determining an initial sub-area to be searched from the current scene includes:

[0020] Determining a key search area from the current scene; wherein the key search area has a higher search priority than other areas in the current scene, and the search priority is used to represent the order in which the mobile device searches for the target object in different areas of the current scene;

[0021] Move from the current position to the key search area, and determine the initial sub-area to be searched in the key search area.

[0022] In an exemplary embodiment of the present disclosure, determining the initial sub-area to be searched in the key search area includes:

[0023] Determining an initial position of the mobile device in the key search area;

[0024] A circular area with the initial position as the center and a preset distance as the radius is determined as the initial sub-area to be found.

[0025] In an exemplary embodiment of the present disclosure, the step of capturing, by the image capturing device, an initial viewfinder image of the mobile device during its movement within the initial sub-area to be found includes:

[0026] Taking the initial position as the rotation center, rotating within the initial sub-area to be found by a preset angle;

[0027] The image acquisition device is used to acquire an initial framing image of the self-moving device during its rotation within the initial sub-area to be found.

[0028] In an exemplary embodiment of the present disclosure, determining a key search area from the current scene includes:

[0029] detecting whether the object search instruction includes a designated search area for the target object;

[0030] If the object search instruction includes the designated search area, the designated search area is determined as the key search area.

[0031] In an exemplary embodiment of the present disclosure, the method further includes:

[0032] If the object search instruction does not include the specified search area, searching historical search records based on the feature description information to obtain the historical appearance area of ​​the target object;

[0033] Determining the historical appearance area as the key search area;

[0034] The historical search record is used to store the mapping relationship between the feature description information of the historical search object and its search result, and the search result includes the historical appearance area of ​​the historical search object.

[0035] In an exemplary embodiment of the present disclosure, after moving from the current position to within the key search area, the method further includes:

[0036] The acquisition frequency of the image acquisition device is controlled to increase from the first frequency to the second frequency.

[0037] In an exemplary embodiment of the present disclosure, after the image acquisition device acquires an initial framing image of the self-mobile device during its movement within the initial sub-area to be found, the method further includes:

[0038] The initial sub-region to be found is marked as an initial found sub-region.

[0039] In an exemplary embodiment of the present disclosure, detecting whether the viewfinder image contains the target object according to the feature description information to obtain an object detection result includes:

[0040] The target object is detected in the initial viewfinder image according to the feature description information to obtain an initial object detection result.

[0041] In an exemplary embodiment of the present disclosure, after obtaining an initial object detection result, determining an object search result for the target object based on the object detection result includes:

[0042] If the initial object detection result indicates that the target object is included in the initial viewfinder image, marking the initial searched sub-region as the appearance region of the target object in a pre-created scene map;

[0043] The marked appearance area and / or the initial view image are output as an object search result for the target object.

[0044] In an exemplary embodiment of the present disclosure, after marking the initial searched sub-region as the appearance region of the target object in the pre-created scene map, the method further includes:

[0045] reshooting the initial framing image in the initial searched sub-region;

[0046] The marked appearance area and / or the retaken initial view image are determined as an object search result for the target object.

[0047] In an exemplary embodiment of the present disclosure, after obtaining an initial object detection result, determining an object search result for the target object based on the object detection result includes:

[0048] If the initial object detection result indicates that the target object is not included in the initial viewfinder image, determining a next sub-region to be searched;

[0049] Capturing, by the image acquisition device, a next framing image of the self-moving device during its movement within the next sub-area to be found;

[0050] Detecting whether the next viewfinder image contains the target object according to the feature description information, and obtaining a next object detection result;

[0051] An object search result for the target object is determined based on the next object detection result.

[0052] In an exemplary embodiment of the present disclosure, determining the next sub-area to be searched includes:

[0053] According to the preset sub-region determination strategy, the next sub-region to be searched is determined;

[0054] Among them, the preset sub-area determination strategy includes: the distance between the center position of the next sub-area to be found and the initial position is less than a preset distance threshold, and the overlapping area between the next sub-area to be found and the initial found sub-area is less than a preset area threshold.

[0055] In an exemplary embodiment of the present disclosure, after determining the next sub-area to be searched, the method further includes:

[0056] Determine the center position of the next sub-region to be found;

[0057] Move from the initial position to the center position of the next sub-area to be found.

[0058] In an exemplary embodiment of the present disclosure, after determining an object search result for the target object based on the object detection result, the method further includes:

[0059] The object search result is pushed to the application and / or user terminal corresponding to the mobile device.

[0060] In an exemplary embodiment of the present disclosure, the target object includes a charging pile, and the object search result includes an appearance area of ​​the charging pile;

[0061] After pushing the object search result to the application and / or user terminal corresponding to the mobile device, the method further includes:

[0062] Go to the appearance area corresponding to the charging pile to charge.

[0063] In an exemplary embodiment of the present disclosure, the object search instruction includes the following form:

[0064] Control instructions issued by the application program on the mobile device;

[0065] Voice control commands;

[0066] A control signal sent by a remote control device of the mobile device;

[0067] The remote control device includes a Bluetooth remote control device, an infrared remote control device and a radio frequency remote control device.

[0068] In an exemplary embodiment of the present disclosure, the target object includes one or more objects;

[0069] When the target object includes multiple objects, the types of any two objects in the multiple objects are the same or different.

[0070] According to a second aspect of the present disclosure, there is provided a self-moving device, wherein the self-moving device is provided with an image acquisition device, and the self-moving device comprises:

[0071] An image acquisition module is configured to acquire, through the image acquisition device, a viewfinder image of the mobile device during movement after receiving an object search instruction, wherein the object search instruction includes feature description information of the target object to be found;

[0072] The object search module is configured to detect whether the viewfinder image contains the target object according to the feature description information, obtain an object detection result, and determine an object search result for the target object based on the object detection result.

[0073] According to a third aspect of the present disclosure, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the object search method described in the first aspect is implemented.

[0074] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the object search method described in the first aspect above by executing the executable instructions.

[0075] As can be seen from the above technical solutions, the object search method, mobile device, computer storage medium, and electronic device in the exemplary embodiments of the present disclosure have at least the following advantages and positive effects:

[0076] In the technical solutions provided in some embodiments of the present disclosure, after receiving an object search instruction (the object search instruction includes feature description information of the target object to be found), a framing image of the mobile device during movement is captured by an image capture device, and whether the framing image contains the target object is detected according to the feature description information to obtain an object detection result, and an object search result for the target object is determined based on the object detection result. On the one hand, this overcomes the limitation problem in the related technology that only objects that have the ability to transmit signals can be identified, expands the types and range of target objects that can be searched, and improves the coverage of the search; on the other hand, the object search result can be quickly output by capturing images and performing image detection based on the feature description information, which not only avoids the time and labor costs caused by manual search for target objects and reduces the burden of manual search, but also can accurately identify the target object with the help of feature description information, reduces the possibility of misidentification, and improves the accuracy of the object search result.

[0077] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0079] FIG1 is a schematic diagram showing a flow chart of an object search method according to an embodiment of the present disclosure;

[0080] FIG2 is a schematic diagram showing a flow chart of how to capture a framing image from a mobile device during movement by an image capture device in an embodiment of the present disclosure;

[0081] FIG3 is a schematic diagram showing how to determine an initial sub-region to be searched in an embodiment of the present disclosure;

[0082] FIG4 is a schematic diagram showing a flow chart of how to determine a key search area in an embodiment of the present disclosure;

[0083] FIG5 is a flowchart illustrating how to determine an object search result based on the initial object detection result when the initial object detection result indicates that the target object is included in the initial viewfinder image in an embodiment of the present disclosure;

[0084] FIG6 is a flowchart illustrating how to determine an object search result based on the initial object detection result when the initial object detection result indicates that the target object is not included in the initial viewfinder image in an embodiment of the present disclosure;

[0085] FIG7 is a schematic diagram showing how to determine the next sub-region to be searched in an embodiment of the present disclosure;

[0086] FIG8 is a schematic diagram showing a flow chart of how a mobile device moves to the next sub-area to be found in an embodiment of the present disclosure;

[0087] FIG9 is a schematic structural diagram of a self-moving device in an exemplary embodiment of the present disclosure;

[0088] FIG10 is a schematic structural diagram of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0089] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0090] The terms "a", "an", "the" and "said" are used in this specification to indicate the presence of one or more elements / components / etc.; the terms "including" and "having" are used to express open-ended inclusion and mean that additional elements / components / etc. may exist in addition to the listed elements / components / etc.; the terms "first" and "second" etc. are used only as labels and are not intended to limit the quantity of their objects.

[0091] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the drawings represent identical or similar parts, and thus repeated descriptions thereof will be omitted. Some of the blocks shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically separate entities.

[0092] In the related art, when a mobile device attempts to return to a pile but fails to receive a pile signal (such as an infrared signal, a pattern, etc.) at a recorded pile coordinate point, the automatic return to pile function cannot be implemented.

[0093] Therefore, the above scheme has at least the following defects: the method of finding objects is single, and the target object must be able to emit a signal or provide an identification mark before it can be found based on the signal. It is too dependent on the pile signal and has great limitations.

[0094] In the embodiments of the present disclosure, firstly, an object search method is provided, which at least to some extent overcomes the defect in the related art that only objects capable of emitting signals can be searched.

[0095] FIG1 is a flow chart showing an object search method according to an embodiment of the present disclosure. The object search method may be executed by a mobile device.

[0096] Referring to FIG1 , an object search method according to an embodiment of the present disclosure includes the following steps:

[0097] Step S110, after receiving the object search instruction, capturing a viewfinder image from the mobile device during movement through an image capture device; the object search instruction includes feature description information of the target object to be found;

[0098] Step S120 , detecting whether the viewfinder image contains a target object according to the feature description information, obtaining an object detection result, and determining an object search result for the target object based on the object detection result.

[0099] In the technical solution provided by the embodiment shown in Figure 1, after receiving an object search instruction (the object search instruction includes feature description information of the target object to be found), the image capture device captures a framing image of the mobile device during movement, detects whether the framing image contains the target object based on the feature description information, obtains an object detection result, and determines the object search result for the target object based on the object detection result. On the one hand, it overcomes the limitation problem of the related technology that only objects with signal transmission capabilities can be identified, expands the types and scope of target objects that can be searched, and improves the coverage of the search; on the other hand, the object search result can be quickly output by collecting images and performing image detection based on feature description information, which not only avoids the time and labor costs caused by manual search for target objects and reduces the burden of manual search, but also can accurately identify the target object with the help of feature description information, reduces the possibility of misidentification, and improves the accuracy of the object search result.

[0100] The following is a detailed description of the specific implementation process of each step in Figure 1:

[0101] Before step S110, it should be noted that the self-moving device (e.g., a sweeping robot) in the present disclosure is provided with an image acquisition device, and the function of the image acquisition device is to capture images or videos. Exemplarily, the image acquisition device can be an RGB camera, which is a sensor that captures images of multiple colors at high speed and identifies and measures the color and brightness of the surface of an object. RGB refers to the three colors of red, green, and blue. The increase and decrease of these three colors constitute a variety of colors, so the RGB camera can capture very rich color images and videos.

[0102] In the present disclosure, a user can control a mobile device to search for a specified target object by issuing an object search instruction.

[0103] The above-mentioned target object can be one or more objects (for example: charging pile, pet, stool, coin, remote control, etc.). When the object contains multiple objects, the types of any two objects in the multiple objects are the same or different, for example: charging pile and pet, pet A (kitten) and pet B (puppy), which can be set according to actual conditions, and this disclosure does not make any special restrictions on this.

[0104] The object search instruction may include feature description information of the target object to be found. The feature description information may include the name, shape, size, color, and frequently placed position of the target object, or may be a picture of the target object. The feature description information can be customized according to the actual situation and is not specifically limited in this disclosure. Given that the feature description information can be flexibly customized, the present disclosure can adapt to the needs of different targets and scenarios, making the solution have broad application prospects.

[0105] The object search instruction may include the following forms:

[0106] ① Control instructions issued by the application of the mobile device, that is, the user can install the application of the mobile device through the Internet or other smart devices. Through the operation interface of the application, the user can input text, pictures, etc. to generate the above control instructions;

[0107] ② Voice control commands, that is, users directly issue commands through voice;

[0108] ③ A control signal sent by a remote control device of the mobile device, wherein the remote control device can be a Bluetooth remote control device, an infrared remote control device, a radio frequency remote control device, etc., which can be set according to actual conditions and is not specifically limited in this disclosure.

[0109] After the user issues the object search instruction, the mobile device may receive the object search instruction, and then step S110 may be executed.

[0110] In step S110 , after receiving the object search instruction, an image capture device captures a framing image of the mobile device during its movement.

[0111] In this step, after receiving the object search instruction, the mobile device may use the image acquisition device to capture the viewfinder image of the mobile device during its movement.

[0112] Specifically, referring to FIG. 2 , FIG. 2 shows a flow chart of how to capture a framing image from a mobile device during movement by an image capture device in an embodiment of the present disclosure, including steps S201 to S202:

[0113] In step S201, an initial sub-region to be found is determined from the current scene.

[0114] In this step, in a first optional implementation, the self-mobile device can directly determine its own initial position in the current scene (which can be the real-time position of the self-mobile device at the moment of receiving the object search instruction, for example: position A), and then use the initial position as the object search starting point to start searching for the target object. Specifically, the self-mobile device can use the initial position as the object search starting point, and can determine a circular area with the initial position as the center and a preset distance as the radius as the initial sub-area to be searched, and start searching for the target object in the initial sub-area to be searched.

[0115] The above-mentioned preset distance length (exemplarily, it can be represented by r) can be determined according to the shooting range or coverable area range of the above-mentioned RGB camera. Exemplarily, the preset distance length r can be any value within the numerical range of 50cm-130cm, and can be set according to actual conditions. This disclosure does not make any special limitations on this.

[0116] Refer to Figure 3, which shows a schematic diagram of how to determine the initial sub-area to be found in an embodiment of the present disclosure. As shown in Figure 3, the shaded circle in the figure represents the initial position of the mobile device, and the circular area containing the shaded circle is the above-mentioned initial sub-area to be found.

[0117] In a second optional implementation, a key search area may be first determined from the current scene, and then the current position may be moved into the key search area to determine an initial sub-area to be searched in the key search area.

[0118] The key search area is the area or range where the target object is most likely to be found. By setting the key search area, the self-mobile device can be guided to quickly find the target object, saving search time and improving object search efficiency. For example, the present disclosure can set the search priority corresponding to the key search area to be higher, while the search priority corresponding to the non-key search area is lower. Thus, the self-mobile device can prioritize searching for the target object in the key search area. If the target object is not found in the key search area, it can continue searching in other non-key search areas or terminate the task to directly generate the object search result.

[0119] Referring to FIG4 , FIG4 shows a schematic flow chart of how to determine a key search area in an embodiment of the present disclosure, including steps S401 to S403:

[0120] In step S401 , it is detected whether the object search instruction includes a designated search area for the target object.

[0121] In this step, the mobile device may detect whether the object search instruction contains a designated search area for the target object. The designated search area may be an area where the target object is often placed or appears.

[0122] In step S402 , if the object search instruction includes a designated search area, the designated search area is determined as a key search area.

[0123] In this step, if it is detected that the object search instruction includes the designated search area, the mobile device may directly determine the designated search area as the key search area.

[0124] In step S403, if the object search instruction does not include a designated search area, historical search records are retrieved according to the feature description information to obtain the historical appearance area of ​​the target object, and the historical appearance area is determined as the key search area.

[0125] In this step, if it is detected that the object search instruction does not include the specified search area, the mobile device may search the historical search record according to the feature description information included in the object search instruction to obtain the historical appearance area of ​​the target object.

[0126] Among them, the above-mentioned historical search records can be used to store the mapping relationship between the feature description information of the historical search object and its search results. The search results can include the historical appearance area of ​​the historical search object, and the historical appearance area is the location area where the target object has appeared in the past or has been frequently placed in the past.

[0127] It should be noted that, assuming that multiple historical appearance areas are retrieved in the historical search record, for example: a, b and c, the mobile device can also count the number of appearances corresponding to each historical appearance area, and then determine the search priority of the above multiple historical appearance areas according to the number of appearances, so as to determine which historical appearance area to search for the above target object first according to the search priority, thereby improving the object search efficiency.

[0128] After determining the key search area, the self-mobile device can move from the current position into the key search area. Thereafter, the self-mobile device can determine its initial position in the key search area (for example, position B). Then, referring to the relevant description of step S201 above, the self-mobile device can use the initial position as the starting point for object search. Specifically, a circular area with the initial position as the center and a preset distance as the radius can be determined as the initial sub-area to be searched, and the search for the target object can be started in the initial sub-area to be searched.

[0129] It should be noted that after moving from the current position to the key search area, the acquisition frequency of the image acquisition device can be controlled to increase from the first frequency to the second frequency, or the image resolution of the image acquisition device can be controlled to increase from the first resolution to the second resolution, or the self-mobile device can also automatically adjust the number of traversals of each sub-area to be searched to two or more times, so as to focus on searching for the above-mentioned target object in the key search area. The specific implementation method can be set according to actual conditions, and the present disclosure does not make any special limitations on this.

[0130] After the initial sub-region to be found is determined, referring to FIG. 2 , in step S202 , an initial framing image of the mobile device during its movement within the initial sub-region to be found is captured by an image capture device.

[0131] In this step, after determining the initial sub-area to be searched, the self-mobile device can rotate a preset angle (the preset angle can be 360 ​​degrees) in the initial sub-area to be searched with the initial position (i.e., position A in the current scene or position B in the key search area) as the rotation center, and capture the initial framing image of the self-mobile device during the rotation process in the initial sub-area to be searched through the image acquisition device.

[0132] After the image acquisition device captures the initial framing image of the mobile device during its rotation in the initial sub-area to be found, the mobile device can mark the initial sub-area to be found as the initial searched sub-area. Based on this, the time-consuming search problem caused by repeated searches of a certain sub-area can be avoided.

[0133] After obtaining the initial framing image, step S120 may be entered to detect whether the framing image contains a target object according to the feature description information, obtain an object detection result, and determine an object search result for the target object based on the object detection result.

[0134] In this step, the mobile device may detect whether the target object is contained in the initial view image based on the feature description information contained in the object search instruction, thereby obtaining an initial object detection result. For example, a pre-trained object recognition model may be pre-deployed in the mobile device. Based on this object recognition model, feature extraction may be performed on the initial view image. The extracted features may then be classified or matched to output an initial object detection result, indicating whether the target object is contained in the initial view image.

[0135] After obtaining the initial object detection result, an object search result for the target object may be determined based on the initial object detection result.

[0136] Specifically, for the case where the above-mentioned initial object detection result indicates that the initial viewfinder image contains the above-mentioned target object, the object search result for the target object can be determined by referring to the processing flow shown in Figure 5; and for the case where the above-mentioned initial object detection result indicates that the initial viewfinder image does not contain the above-mentioned target object, the object search result for the target object can be determined by referring to the processing flow shown in Figure 6.

[0137] First, referring to FIG. 5 , FIG. 5 shows a flowchart of how to determine an object search result based on the initial object detection result when the initial object detection result indicates that the target object is included in the initial viewfinder image in the embodiment of the present disclosure, including steps S501 to S502:

[0138] In step S501 , the initially searched sub-region is marked as the appearance region of the target object in the pre-created scene map.

[0139] In this step, the initially searched sub-area can be marked as the target object's appearance area in a pre-created scene map. The scene map can be a map created indoors by the mobile device using sensors and algorithms before cleaning. This map can include information such as the indoor layout and furniture locations.

[0140] In step S502 , the marked appearance area and / or the initial view image are output as an object search result for the target object.

[0141] In this step, in an optional implementation, the marked appearance area and / or the initial viewfinder image may be output as an object search result for the target object.

[0142] In another optional embodiment, exemplarily, after marking the appearance area of ​​the target object, the shooting resolution of the RGB camera can be increased, and then, the initial framing image can be re-photographed in the initial searched sub-area, and then, the marked appearance area and / or the re-photographed initial framing image can be determined as the object search result for the target object.

[0143] Next, referring to FIG. 6 , FIG. 6 shows a flowchart of how to determine an object search result based on the initial object detection result when the initial object detection result indicates that the initial viewfinder image does not contain the target object, according to an embodiment of the present disclosure. The flowchart includes steps S601 to S604:

[0144] In step S601, the next sub-region to be searched is determined.

[0145] In this step, the self-mobile device can determine the next sub-area to be found from the above-mentioned current scene or the above-mentioned key search area. Exemplarily, the self-mobile device can determine the next sub-area to be found according to a preset sub-area determination strategy, wherein the above-mentioned preset sub-area determination strategy may include: the distance between the center position of the next sub-area to be found and the above-mentioned initial position is less than a preset distance threshold (the preset distance value, for example, can be a value greater than 0 and less than 2r, which can be set according to actual conditions, and the present disclosure does not make special restrictions on this), and the overlapping area between the next sub-area to be found and the above-mentioned initial searched sub-area is less than a preset area threshold (the preset area value can be set according to actual conditions, and the present disclosure does not make special restrictions on this).

[0146] Referring to Figure 7, Figure 7 shows a schematic diagram of how to determine the next sub-area to be found in an embodiment of the present disclosure. As shown in Figure 7, the center position of the next sub-area to be found can be located on the boundary line of the above-mentioned initial searched sub-area, so that the distance between the center position of the next sub-area to be found and the initial position is r, and there is a partial overlapping area between the next sub-area to be found and the initial searched sub-area, so that the self-mobile device can collect the framing image corresponding to the next sub-area to be found by rotating 360 degrees at the center position of the next sub-area to be found. Similarly, the specific method for determining the next sub-area to be found is the same as the method for determining the next sub-area to be found.

[0147] Based on the above strategy, the next sub-area to be searched is determined. The present disclosure enables the self-mobile device to preferentially search the area that is closer to the initial position and has a smaller overlap with the searched area, thereby avoiding repeated invalid searches and unnecessary movements, thereby improving the search efficiency on the one hand; further, considering that the time and power of the self-mobile device are limited, the power allocation of the robot vacuum can be more reasonably utilized, thereby extending the use time of the robot; further, ensuring that the next sub-area to be searched has a certain overlapping area with the searched area, which can avoid the problem of missing blind spots in the search process and improve the integrity of the search.

[0148] After determining the next sub-area to be found, refer to FIG8 , which shows a flow chart of how the mobile device moves to the next sub-area to be found in an embodiment of the present disclosure, including steps S801 and S802:

[0149] In step S801, the center position of the next sub-region to be found is determined.

[0150] In this step, after determining the next sub-region to be found, the mobile device can determine the center position of the next sub-region to be found. The center position can ensure that the mobile device can subsequently rotate 360 ​​degrees to capture images covering every corner of the next sub-region to be found.

[0151] In step S802, the system moves from the initial position to the center position of the next sub-region to be found.

[0152] In this step, after determining the center position of the next sub-region to be found, the mobile device may move from the initial position to the center position of the next sub-region to be found.

[0153] After moving to the center of the next sub-region to be found, referring to FIG. 6 , in step S602 , the image capture device captures the next framing image of the mobile device during its movement within the next sub-region to be found.

[0154] In this step, the self-mobile device can rotate 360 ​​degrees at the center position to capture the next framing image of the self-mobile device during its rotation in the next sub-area to be found through the image capture device. After obtaining the next framing image, the next sub-area to be found can be marked as a found sub-area.

[0155] In step S603, it is detected whether the next viewfinder image contains the target object according to the feature description information to obtain the next object detection result.

[0156] In this step, after obtaining the next viewfinder image, the mobile device can call the above-mentioned object recognition model again, so that the object recognition model detects whether the next viewfinder image contains the target object based on the feature description information contained in the above-mentioned object search instruction, and obtains the next object detection result.

[0157] In step S604 , an object search result for the target object is determined based on the next object detection result.

[0158] In this step, the self-mobile device can determine the object search result for the target object based on the next object detection result. Exemplarily, specifically, when the next object detection result indicates that the target object is included in the next viewfinder image, the next searched sub-area can be directly marked as the appearance area of ​​the target object in the pre-created scene map, and then the marked appearance area and / or the above-mentioned next initial viewfinder image are output as the object search result for the target object. When the next object detection result indicates that the target object is not included in the next viewfinder image, the next sub-area to be searched can be re-determined until the next sub-area to be searched is the last sub-area to be searched in the scene currently located by the self-mobile device, or, when it is detected that the next viewfinder image corresponding to the next sub-area to be searched contains the target object, the search for the target object in the current scene is terminated.

[0159] If the next sub-region to be found is the last sub-region to be found in the scene currently located by the mobile device, and the last viewfinder image corresponding to the last sub-region to be found still does not contain the above-mentioned target object, it can be determined that the search result of the target object is: not found.

[0160] After determining the object search result for the target object based on the object detection result, the present disclosure can push the object search result to the corresponding application and / or user terminal of the mobile device. For example, it can be displayed on the reminder interface of the application, or it can be notified to the user in the form of a voice reply. It can be set according to the actual situation, and the present disclosure does not make any special limitations on this.

[0161] It should be noted that when the above-mentioned target object is a charging pile, if the object search result includes the appearance area of ​​the charging pile, the object search result will be pushed to the corresponding application and / or user terminal of the mobile device, and the mobile device can go to the appearance area corresponding to the charging pile for charging by itself.

[0162] It should also be noted that when the target object in the above-mentioned object search instruction contains two or more objects, illustratively, taking the feature description information of object A, object B and object C contained in the object search instruction as an example, in an optional embodiment, the mobile device can detect whether the viewfinder image contains object A, object B and object C after obtaining each viewfinder image. If any specified object (for example, object A) is contained, the position of the specified object can be marked, and the specified object (object A) can be removed from the objects to be found. Then, the next viewfinder image is collected, and it is detected whether the viewfinder image contains object B and object C, until the above-mentioned three objects are found or the current scene is traversed, the task is terminated, and the object search result is output (it can be a successful result that all three objects are found, or a failed result that none of the three objects are found, or a result that some objects are found and some objects are not found). In another optional embodiment, after obtaining each framed image, the mobile device may first detect whether the framed image contains object A. When object A is found or the current scene is traversed, the task of finding object A is terminated. Thereafter, the mobile device may capture a framed image in the current scene, detect whether the framed image contains object B. When object B is found or the current scene is traversed, the task of finding object B is terminated. Finally, the mobile device may capture a framed image in the current scene, detect whether the framed image contains object C. When object C is found or the current scene is traversed, the task of finding object C is terminated. The above methods can all be set according to actual conditions, and this disclosure does not impose any special restrictions on this.

[0163] Based on the above technical solutions, the present disclosure has at least the following technical effects:

[0164] First, by responding to object search commands to find the target object, the system overcomes the limitation of related technologies that can only identify objects with signal transmission capabilities, expands the types and range of target objects that can be found, and improves the coverage of the search.

[0165] Second, by rotating 360 degrees at each position to capture images within each sub-region to be found, the search continues by selecting the next sub-region closest to the current position and with the smallest overlap area with the current sub-region to be found. This avoids blind spots and ensures that the search area covers every corner of the environment.

[0166] Third, by determining the key search area (i.e., the designated search area of ​​the target object or the historical appearance area of ​​the target object) from the current scene, the target object is searched in the key search area first, which can assist the mobile device to quickly find the target object, save the time of object search, and improve the efficiency of object search.

[0167] The present disclosure further provides a self-mobile device, which is provided with an image acquisition device. FIG9 shows a schematic structural diagram of the self-mobile device in an exemplary embodiment of the present disclosure. As shown in FIG9 , the self-mobile device 900 may include an image acquisition module 910 and an object search module 920. In particular:

[0168] The image acquisition module 910 is configured to acquire, through the image acquisition device, a viewfinder image of the mobile device during movement after receiving an object search instruction including feature description information of the target object to be found;

[0169] The object search module 920 is configured to detect whether the viewfinder image contains the target object according to the feature description information, obtain an object detection result, and determine an object search result for the target object based on the object detection result.

[0170] In an exemplary embodiment of the present disclosure, after receiving the object search instruction, the image acquisition module 910 acquires a viewfinder image of the mobile device during movement through the image acquisition device, including:

[0171] After receiving the object search instruction, determining an initial sub-area to be searched from the current scene;

[0172] The image acquisition device is used to acquire an initial framing image of the self-moving device during its movement within the initial sub-area to be found.

[0173] In an exemplary embodiment of the present disclosure, the image acquisition module 910 determines an initial sub-region to be searched from the current scene, including:

[0174] Determining an initial position of the mobile device in the current scene;

[0175] A circular area with the initial position as the center and a preset distance as the radius is determined as the initial sub-area to be found.

[0176] In an exemplary embodiment of the present disclosure, the image acquisition module 910 determines an initial sub-region to be searched from the current scene, including:

[0177] Determining a key search area from the current scene; wherein the key search area has a higher search priority than other areas in the current scene, and the search priority is used to represent the order in which the mobile device searches for the target object in different areas of the current scene;

[0178] Move from the current position to the key search area, and determine the initial sub-area to be searched in the key search area.

[0179] In an exemplary embodiment of the present disclosure, the image acquisition module 910 determines the initial sub-region to be searched in the key search region, including:

[0180] Determining an initial position of the mobile device in the key search area;

[0181] A circular area with the initial position as the center and a preset distance as the radius is determined as the initial sub-area to be found.

[0182] In an exemplary embodiment of the present disclosure, the image acquisition module 910 acquires an initial view image of the mobile device during its movement within the initial sub-area to be found through the image acquisition device, including:

[0183] Taking the initial position as the rotation center, rotating within the initial sub-area to be found by a preset angle;

[0184] The image acquisition device is used to acquire an initial framing image of the self-moving device during its rotation within the initial sub-area to be found.

[0185] In an exemplary embodiment of the present disclosure, the image acquisition module 910 determines a key search area from the current scene, including:

[0186] detecting whether the object search instruction includes a designated search area for the target object;

[0187] If the object search instruction includes the designated search area, the designated search area is determined as the key search area.

[0188] In an exemplary embodiment of the present disclosure, the image acquisition module 910 is configured to:

[0189] If the object search instruction does not include the specified search area, searching historical search records based on the feature description information to obtain the historical appearance area of ​​the target object;

[0190] Determining the historical appearance area as the key search area;

[0191] The historical search record is used to store the mapping relationship between the feature description information of the historical search object and its search result, and the search result includes the historical appearance area of ​​the historical search object.

[0192] In an exemplary embodiment of the present disclosure, after moving from a current position to within the key search area, the image acquisition module 910 is configured to:

[0193] The acquisition frequency of the image acquisition device is controlled to increase from the first frequency to the second frequency.

[0194] In an exemplary embodiment of the present disclosure, after the image acquisition device acquires an initial framing image of the self-mobile device during its movement within the initial sub-area to be found, the method further includes:

[0195] The initial sub-region to be found is marked as an initial found sub-region.

[0196] In an exemplary embodiment of the present disclosure, the object search module 920 detects whether the viewfinder image contains the target object based on the feature description information, and obtains an object detection result, including:

[0197] The target object is detected in the initial viewfinder image according to the feature description information to obtain an initial object detection result.

[0198] In an exemplary embodiment of the present disclosure, after obtaining an initial object detection result, the object search module 920 determines an object search result for the target object based on the object detection result, including:

[0199] If the initial object detection result indicates that the target object is included in the initial viewfinder image, marking the initial searched sub-region as the appearance region of the target object in a pre-created scene map;

[0200] The marked appearance area and / or the initial view image are output as an object search result for the target object.

[0201] In an exemplary embodiment of the present disclosure, after marking the initial searched sub-region as the appearance region of the target object in the pre-created scene map, the object search module 920 is configured to:

[0202] reshooting the initial framing image in the initial searched sub-region;

[0203] The marked appearance area and / or the retaken initial view image are determined as an object search result for the target object.

[0204] In an exemplary embodiment of the present disclosure, after obtaining an initial object detection result, the object search module 920 determines an object search result for the target object based on the object detection result, including:

[0205] If the initial object detection result indicates that the target object is not included in the initial viewfinder image, determining a next sub-region to be searched;

[0206] Capturing, by the image acquisition device, a next framing image of the self-moving device during its movement within the next sub-area to be found;

[0207] Detecting whether the next viewfinder image contains the target object according to the feature description information, and obtaining a next object detection result;

[0208] An object search result for the target object is determined based on the next object detection result.

[0209] In an exemplary embodiment of the present disclosure, the object search module 920 determines the next sub-region to be searched, including:

[0210] According to the preset sub-region determination strategy, the next sub-region to be searched is determined;

[0211] Among them, the preset sub-area determination strategy includes: the distance between the center position of the next sub-area to be found and the initial position is less than a preset distance threshold, and the overlapping area between the next sub-area to be found and the initial found sub-area is less than a preset area threshold.

[0212] In an exemplary embodiment of the present disclosure, after determining the next sub-region to be searched, the object search module 920 is configured to:

[0213] Determine the center position of the next sub-region to be found;

[0214] Move from the initial position to the center position of the next sub-area to be found.

[0215] In an exemplary embodiment of the present disclosure, after determining an object search result for the target object based on the object detection result, the object search module 920 is configured to:

[0216] The object search result is pushed to the application and / or user terminal corresponding to the mobile device.

[0217] In an exemplary embodiment of the present disclosure, the target object includes a charging pile, and the object search result includes an appearance area of ​​the charging pile;

[0218] After pushing the object search result to the application and / or user terminal corresponding to the mobile device, the object search module 920 is configured to:

[0219] Go to the appearance area corresponding to the charging pile to charge.

[0220] In an exemplary embodiment of the present disclosure, the object search instruction includes the following form:

[0221] Control instructions issued by the application program on the mobile device;

[0222] Voice control commands;

[0223] A control signal sent by a remote control device of the mobile device;

[0224] The remote control device includes a Bluetooth remote control device, an infrared remote control device and a radio frequency remote control device.

[0225] In an exemplary embodiment of the present disclosure, the target object includes one or more objects;

[0226] When the target object includes multiple objects, the types of any two objects in the multiple objects are the same or different.

[0227] The specific details of each module in the above-mentioned mobile device have been described in detail in the corresponding object search method, so they will not be repeated here.

[0228] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0229] Furthermore, although the steps of the method of the present disclosure are described in a particular order in the accompanying drawings, this does not require or imply that the steps must be performed in this particular order, or that all steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0230] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0231] The present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device.

[0232] Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device.

[0233] Computer-readable storage media can transmit, propagate, or transfer programs for use by or in conjunction with an instruction execution system, apparatus, or device. Program code contained on a computer-readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wireline, optical cable, RF, or any suitable combination thereof.

[0234] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiments.

[0235] In addition, an electronic device capable of implementing the above method is also provided in an embodiment of the present disclosure.

[0236] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods, or program products. Therefore, various aspects of the present disclosure may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0237] The electronic device 1000 according to this embodiment of the present disclosure is described below with reference to Figure 10. The electronic device 1000 shown in Figure 10 is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0238] As shown in FIG10 , electronic device 1000 is implemented as a general-purpose computing device. Components of electronic device 1000 may include, but are not limited to, the aforementioned at least one processing unit 1010, the aforementioned at least one storage unit 1020, a bus 1030 connecting various system components (including storage unit 1020 and processing unit 1010), and a display unit 1040.

[0239] The storage unit stores program code, which can be executed by the processing unit 1010, so that the processing unit 1010 performs the steps described in the "Exemplary Method" section of the present specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 1010 can perform the following steps as shown in Figure 1: Step S110, after receiving an object search instruction, captures a viewfinder image from a mobile device during movement through an image capture device; the object search instruction includes feature description information of a target object to be found; Step S120, detects whether the viewfinder image contains the target object based on the feature description information, obtains an object detection result, and determines an object search result for the target object based on the object detection result.

[0240] The storage unit 1020 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 10201 and / or a cache memory unit 10202 , and may further include a read-only memory unit (ROM) 10203 .

[0241] The storage unit 1020 may also include a program / utility 10204 having a set (at least one) of program modules 10205, such program modules 10205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0242] Bus 1030 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0243] The electronic device 1000 can also communicate with one or more external devices 1100 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 1000, and / or any device that enables the electronic device 1000 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 1050. Furthermore, the electronic device 1000 can also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 1060. As shown, the network adapter 1060 communicates with other modules of the electronic device 1000 via a bus 1030. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 1000, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0244] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow from the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

Claims

1. An object searching method, which is applied to a self - moving device. The self - moving device is provided with an image acquisition device. The method includes: After receiving an object searching instruction, acquiring a view - finding image of the self - moving device during the moving process through the image acquisition device; The object searching instruction contains feature description information of a target object to be searched; Detecting whether the view - finding image contains the target object according to the feature description information, obtaining an object detection result, and determining an object searching result for the target object based on the object detection result.

2. The method according to claim 1, wherein The step of, after receiving an object searching instruction, acquiring a view - finding image of the self - moving device during the moving process through the image acquisition device includes: After receiving an object searching instruction, determining an initial sub - area to be searched from the current scene; Acquiring an initial view - finding image of the self - moving device during the moving process within the initial sub - area to be searched through the image acquisition device.

3. The method according to claim 2, wherein The step of determining an initial sub - area to be searched from the current scene includes: Determining an initial position of the self - moving device in the current scene; Determining a circular area with the initial position as the center and a preset distance length as the radius as the initial sub - area to be searched.

4. The method according to claim 2, wherein, The step of determining an initial sub - area to be searched from the current scene includes: Determining a key searching area from the current scene; the searching priority corresponding to the key searching area is higher than that of other areas in the current scene, and the searching priority is used to represent the sequence of the self - moving device searching for the target object in different areas of the current scene; Moving from the current position to within the key searching area, and determining the initial sub - area to be searched within the key searching area.

5. The method according to claim 4, wherein The step of determining the initial sub - area to be searched within the key searching area includes: Determining an initial position of the self - moving device in the key searching area; Determining a circular area with the initial position as the center and a preset distance length as the radius as the initial sub - area to be searched.

6. The method according to any one of claims 2 to 5, wherein The step of, through the image acquisition device, acquiring an initial view - finding image of the self - moving device during the moving process within the initial sub - area to be searched includes: Rotating a preset angle within the initial sub - area to be searched with the initial position as the rotation center; Acquiring an initial view - finding image of the self - moving device during the rotation process within the initial sub - area to be searched through the image acquisition device.

7. The method according to claim 4 or 5, wherein The step of determining a key searching area from the current scene includes: Detecting whether the object searching instruction contains a specified searching area for the target object; If the object searching instruction contains the specified searching area, determining the specified searching area as the key searching area.

8. A self - moving device, which is provided with an image acquisition device. The self - moving device includes: An image acquisition module, configured to, after receiving an object searching instruction, acquire a view - finding image of the self - moving device during the moving process through the image acquisition device; the object searching instruction contains feature description information of a target object to be searched; An object search module, configured to detect whether the captured image contains the target object according to the feature description information, obtain an object detection result, and determine an object search result for the target object based on the object detection result.

9. A computer storage medium, having stored thereon a computer program, which when executed by a processor, implements the object search method according to any one of claims 1 to 7.

10. An electronic device, comprising: a processor; and a memory, configured to store executable instructions of the processor; wherein the processor is configured to execute the object search method according to any one of claims 1 to 7 by executing the executable instructions.

Images