Control instruction generation method and device, storage medium and electronic device

Abstract

The present application discloses a method and device for generating control instructions, a storage medium and an electronic device, wherein the method comprises: performing image acquisition on the lower limbs of a target object to obtain at least two lower limb images; obtaining image recognition results of the at least two lower limb images, wherein the image recognition results are used to indicate at least one of the following: the rotation angle of the lower limbs, the rotation direction of the lower limbs; generating control instructions for controlling a mobile robot according to the image recognition results, and adopting the above technical solution to solve the problem in the related art that, in the scenario of controlling a mobile robot, it is extremely inconvenient to control a mobile robot by gestures.

Description

Technical Field

[0001] The present application relates to the field of communications, and in particular, to a method and device for generating a control instruction, a storage medium, and an electronic device. Background Art

[0002] Mobile robots are machines that automatically perform tasks. They can accept human commands, run pre-programmed programs, or act according to principles formulated using artificial intelligence technology. Such mobile robots can be used indoors or outdoors, in industry or at home, to replace security patrols, to replace people cleaning the floor, to accompany families, to assist in office work, etc.

[0003] Specifically, the cleaning robot can be controlled by a person (for example, an operator holding a remote control) or can complete floor cleaning work in a room according to certain set rules. It can clean up hair, dust, debris, etc. Most existing mobile robots can only accept relatively mechanical control instructions, and the interaction with users is not high, and it is difficult to provide users with a good user experience.

[0004] During the autonomous cleaning process of the mobile robot, if the user temporarily specifies the mobile robot to clean a certain area, or the mobile robot meets the user head-on, in order to make the mobile robot give way to the user, in the related technology, the user can control the mobile robot through gestures. However, when the distance between the mobile robot and the user is close, since the mobile robot lens is facing forward, the user needs to squat down to allow the mobile robot to take pictures. If the user is holding other objects in both hands, he needs to put the objects down and free his hands before gesture control can be performed. Therefore, in the scenario of controlling the mobile robot, it is extremely inconvenient to control the mobile robot through gestures.

[0005] In the related art, in the scenario of controlling a mobile robot, it is extremely inconvenient to control the mobile robot through gestures, and no effective solution has been proposed yet. Summary of the invention

[0006] The embodiments of the present application provide a method and device for generating a control instruction, a storage medium, and an electronic device, so as to at least solve the problem in the related art that it is extremely inconvenient to control a mobile robot by gestures in the scenario of controlling a mobile robot.

[0007] According to one embodiment of the embodiments of the present application, a method for generating control instructions is provided, comprising: performing image acquisition on the lower limb part of a target object to obtain at least two lower limb pictures; obtaining image recognition results of the at least two lower limb pictures, wherein the image recognition results are used to indicate at least one of the following: the rotation angle of the lower limb part, the rotation direction of the lower limb part; and generating a control instruction for controlling a mobile robot according to the image recognition results.

[0008] In an exemplary embodiment, a control instruction for controlling a mobile robot is generated based on the image recognition result, including: determining a time interval for acquiring a first lower limb image and a second lower limb image, wherein the at least two lower limb images include at least: the first lower limb image and the second lower limb image, and the acquisition time of the first lower limb image is earlier than the acquisition time of the second lower limb image; determining a rotation direction of the lower limb part of the target object based on the image recognition result, and determining a rotation time of the lower limb part of the target object based on the time interval; determining a rotation direction of the mobile robot based on the rotation direction of the lower limb part of the target object, and determining a rotation angle of the mobile robot based on the rotation time of the lower limb part of the target object, and generating a control instruction carrying the rotation direction and rotation angle of the mobile robot.

[0009] In an exemplary embodiment, a control instruction for controlling a mobile robot is generated based on the image recognition result, including: determining the rotation direction and rotation angle of the lower limb part of the target object based on the image recognition result; determining the rotation direction of the mobile robot based on the rotation direction of the lower limb part of the target object, and determining the rotation angle of the mobile robot based on the rotation angle of the lower limb part of the target object, and generating a control instruction carrying the rotation angle and rotation direction of the mobile robot.

[0010] In an exemplary embodiment, before generating a control instruction for controlling the mobile robot based on the image recognition result, the method also includes: determining a first feature value of the at least two lower limb images; determining whether the first feature value is consistent with a second feature value of an image of a lower limb part of a preset object; and generating a control instruction for controlling the mobile robot to pause when the first feature value is consistent with the second feature value.

[0011] In an exemplary embodiment, a control instruction for controlling a mobile robot is generated based on the image recognition result, including: determining a first feature value of the at least two lower limb images; determining whether the first feature value is consistent with a third feature value of an image containing preset footwear; and when the first feature value is consistent with the third feature value, generating a control instruction for controlling the mobile robot based on the image recognition result.

[0012] In an exemplary embodiment, a control instruction for controlling a mobile robot is generated based on the image recognition result, comprising: in the case where there are multiple image acquisition devices located on the mobile robot, determining the priority of the multiple image acquisition devices; obtaining at least two third lower limb pictures continuously acquired by the image acquisition device with the highest priority; determining whether a control instruction can be generated based on the image recognition results of the at least two third lower limb pictures; and in the case where it is determined that a control instruction can be generated based on the image recognition results of the at least two third lower limb pictures, determining the control instruction to be generated based on the image recognition results corresponding to the at least two third lower limb pictures.

[0013] In an exemplary embodiment, after determining whether control instructions can be generated based on the image recognition results of the at least two third lower limb images, the method also includes: in case it is determined that control instructions cannot be generated based on the image recognition results of the at least two third lower limb images, obtaining at least two fourth lower limb images of the target object acquired by an image acquisition device with a second highest priority; determining whether control instructions can be generated based on the image recognition results of the at least two fourth lower limb images; in case it is determined that control instructions can be generated based on the image recognition results of the at least two fourth lower limb images, generating corresponding control instructions based on the image recognition results of the at least two fourth lower limb images.

[0014] In an exemplary embodiment, after collecting images of the lower limbs of the target object and obtaining at least two lower limb images, the method further includes: identifying the at least two lower limb images to determine whether other parts exist in any of the at least two lower limb images, wherein the other parts are parts other than the lower limbs of the target object; and when it is determined that other parts exist in any of the at least two lower limb images, encrypting the image containing the other parts in any of the images.

[0015] According to another embodiment of the embodiment of the present application, a control instruction generation device is also provided, including: a first acquisition module, used to collect images of the lower limb part of the target object to obtain at least two lower limb pictures; a second acquisition module, used to obtain image recognition results of the at least two lower limb pictures, wherein the image recognition results are used to indicate at least one of the following: the rotation angle of the lower limb part, the rotation direction of the lower limb part; a generation module, used to generate control instructions for controlling a mobile robot according to the image recognition results.

[0016] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above method when running.

[0017] According to another aspect of the embodiments of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method through the computer program.

[0018] In an embodiment of the present application, images of the lower limb part of the target object are captured to obtain at least two lower limb pictures; image recognition results of the at least two lower limb pictures are obtained, wherein the image recognition results are used to indicate at least one of the following: the rotation angle of the lower limb part, the rotation direction of the lower limb part; control instructions for controlling the mobile robot are generated according to the image recognition results; that is, the mobile robot captures images in real time, and when an image of the lower limb part of the target object is acquired, it is determined that there is a target object near the mobile robot, and then control instructions for controlling the mobile robot are generated according to the image recognition results of at least two lower limb pictures of the target object. The above-mentioned technical solution is adopted to solve the problem that it is extremely inconvenient to control the mobile robot by gestures in the scenario of controlling the mobile robot, and then the technical effect of generating corresponding instructions for controlling the mobile robot by analyzing the lower limb pictures, so as to facilitate the user to control the mobile robot by using lower limb movements. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 It is a hardware structure block diagram of a computer terminal of a method for generating a control instruction according to an embodiment of the present application;

[0021] Figure 2 is a flow chart of a method for generating a control instruction according to an embodiment of the present application;

[0022] Figure 3 is a flow chart of a method for generating a control instruction according to another embodiment of the present application;

[0023] Figure 4 is a lower limb image (I) according to an optional embodiment of the present application;

[0024] Figure 5 is a lower limb image (two) according to an optional embodiment of the present application;

[0025] Figure 6 It is a lower limb picture (three) according to an optional embodiment of the present application;

[0026] Figure 7 is a lower limb image (iv) according to an optional embodiment of the present application;

[0027] Figure 8 is a lower limb image (V) according to an optional embodiment of the present application;

[0028] Fig. 9 is a lower limb image (six) according to an optional embodiment of the present application;

[0029] Fig.10 It is a lower limb picture (VII) according to an optional embodiment of the present application;

[0030] Fig.11 It is a structural block diagram of a control instruction generation device according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] The method embodiments provided in the embodiments of the present application can be executed in a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 1 is a hardware structure block diagram of a computer terminal of a method for generating a control instruction according to an embodiment of the present application. Figure 1 As shown, the computer terminal may include one or more ( Figure 1Only one is shown in the figure) processor 102 (processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. In an exemplary embodiment, the computer terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components as shown, or with Figure 1 Equivalent functions or comparisons shown Figure 1 A different configuration with more features is shown.

[0034] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as computer programs corresponding to the methods in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, that is, the above-mentioned methods are implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0035] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0036] In this embodiment, a method for generating a control instruction is provided, which is applied to a terminal device, specifically, to a mobile robot. Figure 2 : is a flow chart of a method for generating a control instruction according to an embodiment of the present application, and the flow includes the following steps:

[0037] Step S202, collecting images of the lower limbs of the target object to obtain at least two lower limb images;

[0038] It should be noted that it is also possible to collect video of the lower limbs of the target object to obtain video information containing the lower limbs of the target object, and then analyze the video information to obtain the at least two lower limb images.

[0039] It should be noted that the mobile robot is equipped with an image acquisition device, which is located at the front end of the mobile robot in the forward direction, so as to facilitate the control action of photographing and identifying the target object in the forward direction.

[0040] It should be noted that the lower limbs of the target subject include at least one of the following: buttocks, thighs, knees, shins and feet.

[0041] Step S204, obtaining image recognition results of the at least two lower limb images, wherein the image recognition results are used to indicate at least one of the following: a rotation angle of the lower limb part, and a rotation direction of the lower limb part;

[0042] Step S206: generating a control instruction for controlling the mobile robot according to the image recognition result.

[0043] It should be noted that the above-mentioned mobile robots include but are not limited to: sweeping robots and food delivery robots, and the embodiments of the present invention are not limited to this.

[0044] Through the above steps, the lower limb part of the target object is imaged to obtain at least two lower limb pictures; image recognition results of the at least two lower limb pictures are obtained, wherein the image recognition results are used to indicate at least one of the following: the rotation angle of the lower limb part, the rotation direction of the lower limb part; control instructions for controlling the mobile robot are generated according to the image recognition results; that is, the mobile robot collects images in real time, and when the image of the lower limb part of the target object is obtained, it is determined that there is a target object near the mobile robot, and then the control instructions for controlling the mobile robot are generated according to the image recognition results of at least two lower limb pictures of the target object. The above technical solution is adopted to solve the problem that it is extremely inconvenient to control the mobile robot through gestures in the scenario of controlling the mobile robot, and then the technical effect of generating corresponding instructions for controlling the mobile robot by analyzing the lower limb pictures, so as to facilitate the user to control the mobile robot using lower limb movements.

[0045] There are many ways to implement the above step S206. The embodiment of the present invention provides two implementation methods:

[0046] Method 1: Determine the time interval between obtaining a first lower limb image and a second lower limb image, wherein the at least two lower limb images include at least: the first lower limb image and the second lower limb image, and the acquisition time of the first lower limb image is earlier than the acquisition time of the second lower limb image; determine the rotation direction of the lower limb part of the target object according to the image recognition result, and determine the rotation time of the lower limb part of the target object according to the time interval; determine the rotation direction of the mobile robot according to the rotation direction of the lower limb part of the target object, and determine the rotation angle of the mobile robot according to the rotation time of the lower limb part of the target object, and generate a control instruction carrying the rotation direction and rotation angle of the mobile robot.

[0047] It should be noted that the first method is to determine the rotation direction by obtaining the time interval between the first lower limb image and the second lower limb image, for example: there is a corresponding relationship between the time interval and the rotation angle, such as 20° rotation in 1-2 seconds, 40° rotation in 2-4 seconds, 60° rotation in 4-6 seconds, etc.; the rotation direction of the mobile robot is determined by the rotation direction of the lower limb part of the target object, for example: the foot of the target object turns left, and the rotation direction of the mobile robot is right; or the foot of the target object turns left, and the rotation direction of the mobile robot is left. For example: the calf of the target object turns left, and the rotation direction of the mobile robot is right; or the calf of the target object turns left, and the rotation direction of the mobile robot is left.

[0048] It should be noted that obtaining the image recognition results of the at least two lower limb pictures includes: identifying the first lower limb picture and the second lower limb picture; when the recognition result of the first lower limb picture is used to indicate that the front of the lower limb part of the target object is facing the mobile robot, and the recognition result of the second lower limb picture is used to indicate that the side of the lower limb part of the target object is facing the mobile robot, the rotation direction of the lower limb part is determined according to the direction of the side of the lower limb part.

[0049] For example, when the distance between the target object's lower limbs and the ground is less than the preset distance, the robot vacuum cleaner captures the shoe tip (or toe), the shoe side (or foot side), and the shoe heel (or heel part). If the shoe tip is facing the robot vacuum cleaner, then after pausing for t seconds, the shoe side is facing the robot vacuum cleaner. After the robot vacuum cleaner detects and recognizes the shoe tip image, the shoe side, and the shoe tip direction, it determines the rotation angle according to the time interval t, such as 20° in 1-2 seconds, 40° in 2-4 seconds, 60° in 4-6 seconds, etc., and determines the rotation direction of the robot vacuum cleaner according to the direction of the shoe tip when facing the side.

[0050] For example, if the robot vacuum cleaner captures a calf, if the front of the calf faces the robot vacuum cleaner, and then the knee faces the robot vacuum cleaner after a pause of t seconds, the robot vacuum cleaner detects and identifies the front of the calf and the knee, and then determines the rotation angle according to the time interval t, such as 20° in 1-2 seconds, 40° in 2-4 seconds, 60° in 4-6 seconds, etc., and determines the rotation direction of the robot vacuum cleaner according to the direction of the knee.

[0051] Method 2: Determine the rotation direction and rotation angle of the lower limbs of the target object based on the image recognition result; determine the rotation direction of the mobile robot based on the rotation direction of the lower limbs of the target object, and determine the rotation angle of the mobile robot based on the rotation angle of the lower limbs of the target object, and generate a control instruction carrying the rotation angle and rotation direction of the mobile robot.

[0052] It should be noted that the second method is to directly obtain the rotation direction and rotation angle of the lower limbs of the target object based on the lower limb image, and then determine the rotation angle and rotation direction of the mobile robot. For example: the foot of the target object turns 30° to the left, and the mobile robot turns 30° to the right; or the foot of the target object turns 30° to the left, and the mobile robot turns 30° to the left.

[0053] For example, the sweeping robot takes a picture of the front of the calf facing the sweeping robot, and then pauses for t seconds before turning the side of the calf towards the sweeping robot. After the sweeping robot detects and identifies the front and side images of the calf, it determines the rotation angle of the robot based on the rotation angle of the calf. For example, if the calf of the target object turns 30° to the left, the mobile robot turns 30° to the right; or if the calf of the target object turns 30° to the left, the mobile robot turns 30° to the left.

[0054] It should be noted that obtaining the image recognition results of the at least two lower limb pictures includes: determining the rotation angle and rotation direction of the target object according to the first lower limb picture and the second lower limb picture.

[0055] In daily life, there are inevitably children and elderly people at home. For children and the elderly, it may be difficult to control the sweeping robot through lower limb movements. Therefore, before generating the control instructions for controlling the mobile robot according to the image recognition results, the following steps are also included:

[0056] Step 1: Determine the first eigenvalues ​​of the at least two lower limb images;

[0057] Step 2: Determine whether the first characteristic value is consistent with a second characteristic value of the image of the lower limb part of the preset object;

[0058] Step 3: When the first feature value is consistent with the second feature value, generate a control instruction for controlling the mobile robot to pause.

[0059] That is to say, when it is determined that the first eigenvalues ​​of the at least two lower limb images are consistent with the second eigenvalues ​​of the lower limb part of the preset object, it is determined that the target object is an object that does not have the ability to control the sweeping robot, and the moving direction of the target object cannot be estimated. Therefore, a pause control instruction is directly generated to cause the mobile robot to pause. When the distance between the target object and the mobile robot is greater than the preset distance, the mobile robot executes the moving event according to the previously planned path, thereby enhancing the intelligence of the mobile robot.

[0060] For example, when it is determined that the first feature value of the at least two lower limb images is consistent with the second feature value of the image containing the preset lower limb part of a child, the sweeping robot directly suspends work. When the child moves away from the mobile robot, the mobile robot executes the movement event according to the previously planned path.

[0061] In an exemplary embodiment, a control instruction for controlling a mobile robot is generated based on the image recognition result, including: determining a first feature value of the at least two lower limb images; determining whether the first feature value is consistent with a third feature value of an image containing preset footwear; and when the first feature value is consistent with the third feature value, generating a control instruction for controlling the mobile robot based on the image recognition result.

[0062] In order to facilitate the assignment of control instructions to the mobile robot, a mobile client can be used to connect to the mobile robot to send preset instructions to the mobile robot, or permissions can be granted to specified shoes, people, and pants, such as only certain shoes, people, and pants can be controlled. In the case of obtaining the first feature value of the at least two lower limb images, the first feature value is compared with the third feature value of the picture containing the preset footwear; when the first feature value is consistent with the third feature value, a control instruction for controlling the mobile robot is generated according to the image recognition result; when the first feature value is inconsistent with the third feature value, it is determined that the target object has no control authority, and then the control instruction for controlling the mobile robot is not generated according to the image recognition result. In the case of obtaining the first feature value of the at least two lower limb images, the first feature value is compared with the third feature value of the picture containing the preset pants; when the first feature value is consistent with the third feature value, a control instruction for controlling the mobile robot is generated according to the image recognition result; when the first feature value is inconsistent with the third feature value, it is determined that the target object has no control authority, and then the control instruction for controlling the mobile robot is not generated according to the image recognition result.

[0063] In an exemplary embodiment, a control instruction for controlling a mobile robot is generated based on the image recognition result, comprising: in the case where there are multiple image acquisition devices located on the mobile robot, determining the priority of the multiple image acquisition devices; obtaining at least two third lower limb pictures continuously acquired by the image acquisition device with the highest priority; determining whether a control instruction can be generated based on the image recognition results of the at least two third lower limb pictures; and in the case where it is determined that a control instruction can be generated based on the image recognition results of the at least two third lower limb pictures, determining the control instruction to be generated based on the image recognition results corresponding to the at least two third lower limb pictures.

[0064] In order to meet the needs of multiple scenes and expand the range of the mobile robot to shoot and identify the lower limb movements of the target object, cameras can be installed on the front, back, left, and right sides of the mobile robot to increase the recognition range as needed. When multiple image acquisition devices are installed, the priority of the execution of the mobile robot's shooting and identification actions can be set. If the same control action is captured by the front, back, left, and right sides, the priority can be set to the front image acquisition device, the left image acquisition device, the right image acquisition device, and the rear image acquisition device in sequence, that is, only the action corresponding to the image captured by the front image acquisition device is executed. The embodiment of the present invention does not limit the order of priority.

[0065] In an exemplary embodiment, after determining whether control instructions can be generated based on the image recognition results of the at least two third lower limb images, the method also includes: in case it is determined that control instructions cannot be generated based on the image recognition results of the at least two third lower limb images, obtaining at least two fourth lower limb images of the target object acquired by an image acquisition device with a second highest priority; determining whether control instructions can be generated based on the image recognition results of the at least two fourth lower limb images; in case it is determined that control instructions can be generated based on the image recognition results of the at least two fourth lower limb images, generating corresponding control instructions based on the image recognition results of the at least two fourth lower limb images.

[0066] That is to say, the priority can be set in sequence as the front image acquisition device, the left image acquisition device, the right image acquisition device, and the rear image acquisition device. If the control instruction cannot be generated according to the image recognition result of the third lower limb picture obtained according to the highest priority, the control instruction is generated according to the image recognition result of the lower limb picture obtained by the left image acquisition device; if the control instruction cannot be generated according to the image recognition result of the lower limb picture obtained by the left image acquisition device, the control instruction is generated according to the image recognition result of the lower limb picture obtained by the right image acquisition device until the control instruction is generated.

[0067] In an exemplary embodiment, after collecting images of the lower limbs of the target object and obtaining at least two lower limb images, the method further includes: identifying the at least two lower limb images to determine whether other parts exist in any of the at least two lower limb images, wherein the other parts are parts other than the lower limbs of the target object; and when it is determined that other parts exist in any of the at least two lower limb images, encrypting the image containing the other parts in any of the images.

[0068] That is to say, in order to ensure that the user's privacy information is not leaked, the mobile robot processes the collected images, analyzes only the lower limbs, encrypts / removes private parts such as the face, and thus achieves the technical effect of not leaking sensitive privacy information such as the face.

[0069] In order to better understand the process of the above method, the above implementation method flow is described below in combination with an optional embodiment, but it is not used to limit the technical solution of the embodiment of the present application.

[0070] In the embodiment of the present invention, when the user uses the lower limbs to control the sweeper, that is, the user can use the hips, thighs, knees, shins and feet to control the sweeper. Regardless of whether the user is barefoot / socks, or wearing shoes. In addition, when the user wears shoes, the shoes are photographed by the front end of the sweeper. The upper and the sole are different. For example, when standing, the side of the shoe tip is Figure 5 As shown, the positive side is Figure 6 As shown in the figure, when the sole of the shoe is facing the sweeper, Figure 7 As shown, the heel of the shoe Figure 8 shown.

[0071] In this embodiment, a method for generating a control instruction is provided. Figure 3 is a flow chart of a method for generating a control instruction according to another embodiment of the present application, such as Figure 3 As shown, the specific steps are as follows:

[0072] Step S301: when the sweeping machine (equivalent to the mobile robot in the above embodiment) captures a motion picture, the motion instruction recognition function is started;

[0073] Step S302: extracting feature values ​​of a plurality of images (equivalent to at least two lower limb images in the above embodiment) of the collected feet (equivalent to the lower limb part in the above embodiment);

[0074] Step S303: comparing the feature value of each image with the preset feature value to determine the recognition result of each image;

[0075] Step S304: determining the image combination recognition result corresponding to the plurality of images (equivalent to the image recognition result in the above embodiment) according to the recognition result of each image;

[0076] The feature values ​​of the captured images are extracted, and then compared with the preset feature values, and the pictures are connected together to form an image combination recognition result. Then, it is queried whether there is a corresponding control instruction in the image combination recognition result. If so, it is executed. If not, it continues to proceed according to the original cleaning plan and route.

[0077] Step S305: determining a control instruction according to the image recognition result;

[0078] For example, in an optional embodiment, when the user is standing, the robot vacuum cleaner can only capture the toe, side, and heel of the shoe. If the toe of the shoe is facing the robot vacuum cleaner, after a pause of t seconds, the user turns the side of the shoe toward the robot vacuum cleaner. The robot vacuum cleaner detects and recognizes the toe image, the side of the shoe, and the direction of the toe, and determines the rotation angle based on the time t, such as 20° in 1-2 seconds, 40° in 2-4 seconds, 60° in 4-6 seconds, etc., and determines the direction of the robot vacuum cleaner based on the direction of the toe of the shoe when facing the side. Figure 8 As shown, Figure 8 This is the lower limb picture (five) according to the optional embodiment of the present application. After the sweeping robot captures the toe and side of the shoe, it determines the sweeping direction and angle according to the time interval t and the direction of the shoe, that is, turns 40° to the left.

[0079] For example, in an optional embodiment, if the user lifts his foot so that the sole is facing the camera of the sweeper, or sits on the sofa with the sole facing the camera of the sweeper, and then turns the toe of the shoe to face the camera of the sweeper, the sweeper analyzes the sole feature value and steering angle of the two images, as well as the deviation of the toe direction. Fig. 9 and Fig.10 As shown, Fig. 9 is a lower limb image (six) according to an optional embodiment of the present application; Fig.10 is a lower limb image (VII) according to an optional embodiment of the present application, Fig. 9 and Fig.10 It is a picture of the sole control action facing the sweeper. The characteristic values ​​of the two pictures are used to determine whether they are pictures of the same sole. If they are pictures of the same sole, the angle between the two pictures is calculated. According to the angle and the toe deviation, the sweeper turns left N degrees. If there is no subsequent action instruction, the cleaning route is replanned. As long as there is an angle between the two pictures, it is not required that the sole is facing the sweeper camera vertically. Since the shoe control will not be so precise, the angle preset to be less than 10° can be considered as jitter and is not processed.

[0080] Step S306: executing the control instruction and continuously monitoring;

[0081] Step S307: If there is no subsequent instruction action, re-plan the cleaning route.

[0082] Optionally, the sweeper is equipped with a camera, which is located at the front end of the sweeper in the forward direction, so as to capture and identify the user's control actions in the forward direction. In order to meet the needs of multiple scenarios and expand the range of the sweeper to capture and identify the user's footsteps, cameras can be installed on the front, back, left, and right sides of the sweeper as needed to increase the recognition range. When multiple cameras are installed, the priority of the sweeper's capture and identification actions can be set. If the same control action is captured by the front, back, left, and right sides, the priority can be set to front, left, right, and back in sequence.

[0083] Optionally, if a human figure / face is captured, the video is analyzed and only the area below the ankle is retained for analysis, and images of non-foot areas are subjected to privacy masking or video encryption.

[0084] Optionally, in order to facilitate the assignment of control instructions to the sweeping machine, the mobile client can be used to connect to the sweeping machine to preset control instructions for the sweeping machine, or permissions can be granted to specified shoes, such as only a certain pair of shoes can be controlled. Fig. 9 and Fig.10 As shown, set the corresponding control instructions of the sweeper through the mobile APP setting page, such as Figure 5-Figure 8 , when the user presses the picture on the mobile app, stand and put the toe of the shoe in front of the camera of the sweeper to take picture 5; when the user presses the picture on the mobile app, stand and put the side of the shoe in front of the camera of the sweeper to take picture 6; when the user presses the picture on the mobile app, stand and put the sole of the shoe in front of the camera of the sweeper to take picture 7; when the user presses the picture on the mobile app, stand and put the heel of the shoe in front of the camera of the sweeper to take picture 8. After the subsequent sweeper takes these two consecutive pictures, the left turn instruction corresponding to the left sweeper can be executed. For the convenience of users, when the user presses the video on the mobile app, the user can shoot a continuous action video in front of the sweeper camera, such as left turn 2, and record a video of the right shoe toe turning to the side of the shoe (the shoe toe facing left) when standing. The pause time of 2 seconds represents a turn of 20°. When the subsequent sweeper recognizes the above continuous actions (the right shoe toe turning to the side of the shoe (the shoe toe facing left)), it can execute the corresponding left turn 20° instruction. As shown in Table 1, the same sweeper command can correspond to multiple control actions. To facilitate the control of the sweeper, it is not limited to other parameters such as the time interval t between images or the angle after the same image moves. If it is video control, only one video is recorded.

[0085] When the sweeper leaves the factory, the foot gestures corresponding to the sweeper instructions can be preset. As long as the sweeper camera recognizes the shoe outline or foot outline, the embodiment of the present invention can recommend the user which control instruction is easy to remember, which saves the user from the tedious setting. The user can also set the action instruction or grant control authority to the specified shoe or foot.

[0086] Table 1

[0087]

[0088] In an embodiment of the present invention, a method for processing a captured image by a sweeping machine is provided. By processing the captured human figure / face image, only shoe / foot movements are extracted to protect privacy and prevent sensitive information above the feet, such as faces, from being leaked when at home. By analyzing foot gesture images and videos, the corresponding instructions of the sweeping machine are controlled, making it convenient for users to control the sweeping machine using footstep movements. By connecting to the mobile terminal of the sweeping machine to set the corresponding image or video of the command, the user's accustomed foot gesture control instructions can be set, making it convenient for users to independently set the sweeping machine control commands. It can better perform human-computer interaction and serve different family members in the family.

[0089] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0090] Fig.11 is a structural block diagram of a control instruction generating device according to an embodiment of the present application; Fig.11 As shown, including:

[0091] The first acquisition module 1102 is used to collect images of the lower limbs of the target object to obtain at least two lower limb images;

[0092] The second acquisition module 1104 is used to acquire the image recognition results of the at least two lower limb pictures, wherein the image recognition results are used to indicate at least one of the following: the rotation angle of the lower limb part, the rotation direction of the lower limb part;

[0093] The generating module 1106 is used to generate a control instruction for controlling the mobile robot according to the image recognition result.

[0094] Through the above-mentioned device, images of the lower limbs of the target object are collected to obtain at least two lower limb pictures; image recognition results of the at least two lower limb pictures are obtained, wherein the image recognition results are used to indicate at least one of the following: the rotation angle of the lower limb part, the rotation direction of the lower limb part; control instructions for controlling the mobile robot are generated according to the image recognition results; that is, the mobile robot collects images in real time, and when the image of the lower limb part of the target object is obtained, it is determined that there is a target object near the mobile robot, and then the control instructions for controlling the mobile robot are generated according to the image recognition results of at least two lower limb pictures of the target object collected. The above-mentioned technical solution is adopted to solve the problem that it is extremely inconvenient to control the mobile robot through gestures in the scenario of controlling the mobile robot, and then the technical effect of generating corresponding instructions for controlling the mobile robot by analyzing the lower limb pictures, so as to facilitate the user to control the mobile robot using lower limb movements.

[0095] In an exemplary embodiment, a generation module 1106 is used to determine the time interval between acquiring a first lower limb image and a second lower limb image, wherein the at least two lower limb images include at least: the first lower limb image and the second lower limb image, and the acquisition time of the first lower limb image is earlier than the acquisition time of the second lower limb image; determine the rotation direction of the lower limb part of the target object according to the image recognition result, and determine the rotation time of the lower limb part of the target object according to the time interval; determine the rotation direction of the mobile robot according to the rotation direction of the lower limb part of the target object, and determine the rotation angle of the mobile robot according to the rotation time of the lower limb part of the target object, and generate a control instruction carrying the rotation direction and rotation angle of the mobile robot.

[0096] In an exemplary embodiment, the generation module 1106 is used to determine the rotation direction and rotation angle of the lower limb part of the target object based on the image recognition result; determine the rotation direction of the mobile robot based on the rotation direction of the lower limb part of the target object, and determine the rotation angle of the mobile robot based on the rotation angle of the lower limb part of the target object, and generate a control instruction carrying the rotation angle and rotation direction of the mobile robot.

[0097] In an exemplary embodiment, a generation module 1106 is used to determine a first feature value of the at least two lower limb images; determine whether the first feature value is consistent with a second feature value of an image of a lower limb part of a preset object; and generate a control instruction for controlling the mobile robot to pause when the first feature value is consistent with the second feature value.

[0098] In an exemplary embodiment, generation module 1106 is used to determine a first feature value of the at least two lower limb images; determine whether the first feature value is consistent with a third feature value of a picture containing preset footwear; and when the first feature value is consistent with the third feature value, generate a control instruction for controlling the mobile robot according to the image recognition result.

[0099] In an exemplary embodiment, generation module 1106 is used to determine the priorities of multiple image acquisition devices when there are multiple image acquisition devices located on the mobile robot; obtain at least two third lower limb images continuously acquired by the image acquisition device with the highest priority; determine whether control instructions can be generated based on the image recognition results of the at least two third lower limb images; and when it is determined that control instructions can be generated based on the image recognition results of the at least two third lower limb images, determine the control instructions to be generated based on the image recognition results corresponding to the at least two third lower limb images.

[0100] In an exemplary embodiment, generation module 1106 is used to obtain at least two fourth lower limb images of the target object acquired by the second highest priority image acquisition device when it is determined that control instructions cannot be generated based on the image recognition results of the at least two third lower limb images; determine whether control instructions can be generated based on the image recognition results of the at least two fourth lower limb images; and generate corresponding control instructions based on the image recognition results of the at least two fourth lower limb images when it is determined that control instructions can be generated based on the image recognition results of the at least two fourth lower limb images.

[0101] In an exemplary embodiment, the above-mentioned device also includes: an encryption module, used to identify the at least two lower limb images to determine whether there are other parts in any of the at least two lower limb images, wherein the other parts are parts other than the lower limb parts of the target object; when it is determined that there are other parts in any of the at least two lower limb images, encrypting the image containing the other parts in any of the images.

[0102] An embodiment of the present application further provides a storage medium, which includes a stored program, wherein the program executes any of the above methods when it is run.

[0103] Optionally, in this embodiment, the storage medium may be configured to store program codes for executing the following steps:

[0104] S1, collecting images of the lower limbs of the target object to obtain at least two lower limb images;

[0105] S2, obtaining image recognition results of the at least two lower limb images, wherein the image recognition results are used to indicate at least one of the following: a rotation angle of the lower limb part, and a rotation direction of the lower limb part;

[0106] S3, generating a control instruction for controlling the mobile robot according to the image recognition result.

[0107] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0108] In some embodiments, the electronic device may be a mobile robot, such as a sweeping robot.

[0109] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0110] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0111] S1, collecting images of the lower limbs of the target object to obtain at least two lower limb images;

[0112] S2, obtaining image recognition results of the at least two lower limb images, wherein the image recognition results are used to indicate at least one of the following: a rotation angle of the lower limb part, and a rotation direction of the lower limb part;

[0113] S3, generating a control instruction for controlling the mobile robot according to the image recognition result.

[0114] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0115] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0116] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0117] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating a control instruction, characterized in that: include: Capturing images of the lower limbs of the target object to obtain at least two lower limb images; Obtaining image recognition results of the at least two lower limb images, wherein the image recognition results are used to indicate at least one of the following: a rotation angle of the lower limb part, a rotation direction of the lower limb part; generating a control instruction for controlling the mobile robot according to the image recognition result; Among them, generating control instructions for controlling the mobile robot according to the image recognition result includes: determining the first feature value of the at least two lower limb images; determining whether the first feature value is consistent with the third feature value of the image containing the preset footwear; when the first feature value is consistent with the third feature value, generating control instructions for controlling the mobile robot according to the image recognition result.

2. The method for generating a control instruction according to claim 1, characterized in that: Generating a control instruction for controlling the mobile robot according to the image recognition result, including: Determine a time interval between obtaining a first lower limb picture and a second lower limb picture, wherein the at least two lower limb pictures at least include: the first lower limb picture and the second lower limb picture, and the first lower limb picture is obtained before the second lower limb picture; Determining a rotation direction of the lower limb of the target object according to the image recognition result, and determining a rotation time of the lower limb of the target object according to the time interval; The rotation direction of the mobile robot is determined according to the rotation direction of the lower limbs of the target object, and the rotation angle of the mobile robot is determined according to the rotation time of the lower limbs of the target object, and a control instruction carrying the rotation direction and rotation angle of the mobile robot is generated.

3. The method for generating a control instruction according to claim 1, characterized in that: Generating a control instruction for controlling the mobile robot according to the image recognition result, including: Determine the rotation direction and rotation angle of the lower limb of the target object according to the image recognition result; The rotation direction of the mobile robot is determined according to the rotation direction of the lower limbs of the target object, and the rotation angle of the mobile robot is determined according to the rotation angle of the lower limbs of the target object, and a control instruction carrying the rotation angle and rotation direction of the mobile robot is generated.

4. The method for generating a control instruction according to claim 1, characterized in that: Before generating a control instruction for controlling the mobile robot according to the image recognition result, the method further includes: Determining first feature values ​​of the at least two lower limb images; Determining whether the first characteristic value is consistent with a second characteristic value of the image containing the lower limb part of the preset object; When the first feature value is consistent with the second feature value, a control instruction for controlling the mobile robot to pause is generated.

5. The method for generating a control instruction according to claim 1, characterized in that: Generating a control instruction for controlling the mobile robot according to the image recognition result, including: In the case where there are multiple image acquisition devices on the mobile robot, determining the priorities of the multiple image acquisition devices; Acquire at least two third lower limb images continuously acquired by the image acquisition device with the highest priority; determining whether a control instruction can be generated according to the image recognition results of the at least two third lower limb images; In a case where it is determined that a control instruction can be generated based on the image recognition results of the at least two third lower limb pictures, the generated control instruction is determined based on the image recognition results corresponding to the at least two third lower limb pictures.

6. The method for generating a control instruction according to claim 5, characterized in that: After determining whether a control instruction can be generated according to the image recognition results of the at least two third lower limb pictures, the method further includes: When it is determined that the control instruction cannot be generated according to the image recognition results of the at least two third lower limb images, acquiring at least two fourth lower limb images of the target object acquired by the image acquisition device with the second highest priority; determining whether a control instruction can be generated according to the image recognition results of the at least two fourth lower limb pictures; In a case where it is determined that a control instruction can be generated based on the image recognition results of the at least two fourth lower limb pictures, a corresponding control instruction is generated based on the image recognition results of the at least two fourth lower limb pictures.

7. The method for generating a control instruction according to claim 1, characterized in that: After collecting images of the lower limbs of the target object to obtain at least two lower limb images, the method further includes: Identify the at least two lower limb images to determine whether any one of the at least two lower limb images contains other parts, wherein the other parts are parts other than the lower limb parts of the target object; When it is determined that any one of the at least two lower limb images contains other parts, the image containing the other parts in any one of the images is encrypted.

8. A control instruction generating device, characterized in that: include: A first acquisition module is used to collect images of the lower limbs of the target object to obtain at least two lower limb images; A second acquisition module is used to acquire image recognition results of the at least two lower limb pictures, wherein the image recognition results are used to indicate at least one of the following: a rotation angle of the lower limb part, and a rotation direction of the lower limb part; A generating module, used for generating a control instruction for controlling the mobile robot according to the image recognition result; Among them, the generation module is also used to determine the first feature value of the at least two lower limb images; determine whether the first feature value is consistent with the third feature value of the image containing the preset footwear; when the first feature value is consistent with the third feature value, generate a control instruction for controlling the mobile robot according to the image recognition result.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 7 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 through the computer program.

Images