Device control method, device and computer program product

Abstract

The invention is suitable for the technical field of equipment control, and provides an equipment control method, equipment and a computer program product. The equipment control method is applied to target equipment provided with a three-dimensional distance sensor and a shooting device, and comprises the following steps: acquiring a depth image and a reflectivity image of an environment where the target equipment is located through the three-dimensional distance sensor; obtaining a texture image of an environment where the target equipment is located through a shooting device; according to the depth image, the reflectivity image and the texture image, determining the object type of a target obstacle in the environment where the target equipment is located; and controlling the target equipment to move according to the object type. According to the equipment control method, the omnibearing information of the target obstacle can be fully acquired according to the depth image, the reflectivity image and the texture image of the environment where the target equipment is located, and then the target equipment is controlled to move, so that the obstacle avoidance accuracy of the target equipment is improved.

Description

Technical Field

[0001] This application belongs to the field of equipment control technology, and in particular relates to an equipment control method, equipment, and computer program product. Background Technology

[0002] To enable the target device to avoid obstacles in its environment during movement, the first method involves setting up a camera on the target device to acquire texture images of the obstacles and then using these images to determine obstacle avoidance information. However, this method struggles to obtain spatial information about the obstacles, making it impossible to determine their height and depth, thus reducing the accuracy of obstacle avoidance.

[0003] The second method involves setting up a lidar on the target device and using the lidar to obtain the location information of the target obstacle, and then avoiding obstacles based on the location information of the target obstacle. However, this method has difficulty obtaining information on hollow structures (such as the space structure enclosed by the legs and tabletop of a table) and ramp structures (such as ramp stairs) of the target obstacle, and it is also difficult to obtain information on target obstacles with low reflectivity, which reduces the obstacle avoidance accuracy of the target device.

[0004] It can be seen that the existing obstacle avoidance methods for target devices have low accuracy. Summary of the Invention

[0005] In view of this, embodiments of this application provide a device control method, device, and computer program product to solve the technical problem of low accuracy in existing obstacle avoidance methods for target devices.

[0006] In a first aspect, embodiments of this application provide a device control method applied to a target device equipped with a three-dimensional distance sensor and an imaging device, the method comprising:

[0007] The three-dimensional distance sensor is used to acquire depth and reflectance images of the environment in which the target device is located.

[0008] The imaging device acquires a texture image of the environment in which the target device is located.

[0009] Based on the depth image, the reflectivity image, and the texture image, determine the object type of the target obstacle in the environment where the target device is located;

[0010] The movement of the target device is controlled according to the type of object.

[0011] Optionally, determining the object type of the target obstacle in the environment where the target device is located based on the depth image, the reflectivity image, and the texture image includes:

[0012] If the depth information of the target obstacle in the depth image changes abruptly, and the reflectivity of the target obstacle determined according to the reflectivity image is lower than a first threshold, then the object type is determined to be a transparent object.

[0013] If the depth information of the target obstacle in the depth image does not change abruptly, and the reflectivity of the target obstacle determined according to the reflectivity image is higher than or equal to the first threshold and lower than the second threshold, then the object type is determined to be a low reflectivity object.

[0014] If the reflectance of the target obstacle determined based on the reflectance image is higher than or equal to the second threshold and lower than the third threshold, then the object type is determined to be a medium reflectance object, and the subtype of the medium reflectance object is determined based on the texture image.

[0015] If the reflectivity of the target obstacle determined based on the reflectivity image is higher than or equal to the third threshold, then the object type is determined to be a high-reflectivity object.

[0016] Optionally, controlling the movement of the target device according to the object type includes:

[0017] If the object type is any of the transparent object, the low reflectivity object, or the high reflectivity object, then the safe distance of the target obstacle is determined according to the object type, and different object types correspond to different safe distances;

[0018] If the object type is a medium reflectivity object, then the safe distance of the target obstacle is determined according to the subtype, and different subtypes correspond to different safe distances;

[0019] The target device is controlled to move based on the safe distance and the current distance between the target device and the target obstacle.

[0020] Optional, also includes:

[0021] Based on the depth image, the reflectivity image, and the texture image, the location and structural information of the target obstacle are determined; the structural information includes any one or more of height, width, and length information.

[0022] If the object type is any of the transparent object, the low reflectivity object, or the high reflectivity object, then the target obstacle is marked in a preset electronic map according to the object type, the location information, and the structural information;

[0023] If the object type is a medium reflectivity object, then the target obstacle is marked in a preset electronic map according to the subtype, the location information, and the structural information.

[0024] Optionally, controlling the movement of the target device according to the object type includes:

[0025] Based on the depth image, determine whether the target obstacle includes a hollow structure;

[0026] When the target obstacle includes the hollow structure, the movement of the target device is controlled according to the structural information of the hollow structure and the type of the object.

[0027] Optionally, controlling the movement of the target device according to the object type includes:

[0028] Based on the depth image, determine whether the target obstacle includes a ramp structure;

[0029] When the target obstacle includes the ramp structure, the movement of the target device is controlled according to the structural information of the ramp structure and the type of the object.

[0030] Optionally, the three-dimensional distance sensor and the imaging device are coaxially mounted on the target device; before determining the object type of the target obstacle in the environment of the target device based on the depth image, the reflectivity image, and the texture image, the method further includes:

[0031] Based on the timestamps corresponding to the depth image, reflectance image, and texture image, time alignment processing is performed on the depth image, reflectance image, and texture image;

[0032] Spatial alignment processing is performed on the depth image, the reflectivity image, and the texture image based on the rotation matrix and translation vector between the three-dimensional distance sensor and the imaging device.

[0033] Optional, also includes:

[0034] The environmental complexity of the environment in which the target device is located is determined based on the depth image, the reflectivity image, and the texture image.

[0035] The operating parameters of the three-dimensional distance sensor are adjusted according to the environmental complexity; wherein the operating parameters include image resolution, and the relationship between environmental complexity and image resolution is positively correlated.

[0036] In a second aspect, embodiments of this application provide a target device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the device control method as described in any of the first aspects above.

[0037] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the device control method as described in any of the first aspects above.

[0038] Fourthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to perform the steps of the device control method as described in any of the first aspects above.

[0039] The device control method, device, and computer program product provided in this application have the following beneficial effects:

[0040] The device control method provided in this application is applied to a target device equipped with a three-dimensional distance sensor and an imaging device. First, the three-dimensional distance sensor acquires depth and reflectance images of the environment in which the target device is located, and the imaging device acquires texture images of the environment. Then, based on the depth, reflectance, and texture images, the object type of the target obstacle in the environment is determined. Finally, the movement of the target device is controlled according to the object type. This device control method can fully acquire omnidirectional information about the target obstacle based on the depth, reflectance, and texture images of the environment before controlling the movement of the target device. Compared to existing technologies, this device control method can acquire not only spatial information of the target obstacle but also information about target obstacles with low reflectance before controlling the movement of the target device. Therefore, this device control method can improve the obstacle avoidance accuracy of the target device. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A flowchart illustrating the implementation of the device control method provided in this application embodiment;

[0043] Figure 2This is a schematic diagram of the structure of a device control apparatus provided in an embodiment of this application;

[0044] Figure 3 This is a schematic diagram of the structure of a target device provided in an embodiment of this application. Detailed Implementation

[0045] It should be noted that the terminology used in the embodiments of this application is only for explaining specific embodiments of this application and is not intended to limit this application. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, "at least one" or "one or more" means one, two or more. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0046] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0047] The device control method provided in this application can be executed by a target device equipped with a three-dimensional distance sensor and a camera. For example, the target device may include, but is not limited to, a robotic vacuum cleaner equipped with a three-dimensional distance sensor and a camera.

[0048] The device control method provided in this application can be applied to any scenario where motion control of a target device is required. For example, the target device can be a robotic vacuum cleaner equipped with a three-dimensional distance sensor and a camera. When the robotic vacuum cleaner is performing its sweeping work, the various steps of the device control method provided in this application can be executed by the robotic vacuum cleaner, thereby enabling the robotic vacuum cleaner to avoid obstacles more accurately during the sweeping process.

[0049] Please see Figure 1 , Figure 1This is a flowchart illustrating the implementation of the device control method provided in this application embodiment. This device control method can be applied to a target device equipped with a three-dimensional distance sensor and an imaging device. The device control method may include steps S101 to S104, detailed below:

[0050] In S101, a three-dimensional distance sensor is used to acquire depth and reflectivity images of the environment in which the target device is located.

[0051] In this embodiment of the application, the target device can acquire depth and reflectance images of the environment in which the target device is located using a three-dimensional distance sensor with preset operating parameters.

[0052] In one possible implementation, the three-dimensional distance sensor can be a three-dimensional time-of-flight (3dtof) sensor.

[0053] Among them, the depth image can be used to describe the distance from any point in the environment where the target device is located to the three-dimensional distance sensor; the reflectance image can be used to describe the light reflectance of the object corresponding to any point in the environment where the target device is located.

[0054] In S102, a texture image of the environment in which the target device is located is acquired by the imaging device.

[0055] In one possible implementation, the imaging device can be a two-dimensional RGB camera, based on which the texture image can be a color 2D image.

[0056] In one possible implementation, the 3D distance sensor and the imaging device can be coaxially mounted on the target device. Specifically, the 3D distance sensor and the imaging device can be mounted side-by-side on the target device and fixed by a precision bracket, ensuring that the optical axes of the lenses of the 3D distance sensor and the imaging device remain parallel, and minimizing the distance between the center points of the 3D distance sensor and the imaging device. Ultimately, this allows the depth image, reflectivity image, and texture image to be aligned spatially as much as possible.

[0057] Optionally, after acquiring the depth image, reflectance image, and texture image, the target device can perform time alignment processing on the depth image, reflectance image, and texture image based on their respective timestamps. Specifically, the target device can embed a timestamp into each depth image, reflectance image, and texture image, and then perform time alignment processing on each depth image, reflectance image, and texture image based on their respective timestamps.

[0058] Optionally, spatial alignment processing can be performed on the depth image, reflectance image, and texture image based on the rotation matrix and translation vector between the 3D distance sensor and the imaging device. Specifically, coordinate transformations can be performed on the depth image, reflectance image, and texture image respectively based on the rotation matrix and translation vector between the 3D distance sensor and the imaging device, thereby aligning the depth image, reflectance image, and texture image in coordinates.

[0059] By performing temporal and spatial alignment processing on depth images, reflectivity images, and texture images, the obstacle avoidance accuracy of target devices can be further improved.

[0060] In S103, the object type of the target obstacle in the environment where the target device is located is determined based on the depth image, reflectance image, and texture image.

[0061] In this embodiment of the application, after acquiring the depth image, reflectance image, and texture image, the target device can determine the object type of the target obstacle in the environment in which the target device is located based on the depth image, reflectance image, and texture image through steps a to d, as detailed below:

[0062] In step a, if the depth information of the target obstacle in the depth image changes abruptly, and the reflectivity of the target obstacle determined from the reflectivity image is lower than the first threshold, then the object type is determined to be a transparent object.

[0063] For example, the target device can determine a sudden change in the depth information of a target obstacle in the depth image by the following method: for any two consecutive locations of the target obstacle in the depth image, if the difference between the distances corresponding to the two consecutive locations and the three-dimensional distance sensor is greater than a preset distance threshold, then it can be determined that a sudden change in the depth information of the target obstacle in the depth image has occurred. The preset distance threshold can be set according to actual needs.

[0064] For example, the reflectivity of the target obstacle can be the average of the reflectivity at each location point of the target obstacle; the first threshold can be 5%, meaning the target device can determine the object type of the target obstacle with a reflectivity of less than 5% as a transparent object. In practical applications, transparent objects can include, but are not limited to, glass and transparent plastics.

[0065] In step b, if the depth information of the target obstacle in the depth image does not change abruptly, and the reflectivity of the target obstacle determined from the reflectivity image is higher than or equal to the first threshold and lower than the second threshold, then the object type is determined to be a low reflectivity object.

[0066] For example, the target device can determine that there is no abrupt change in the depth information of the target obstacle in the depth image by the following method: if the difference in distance between any two consecutive locations of the target obstacle in the depth image is less than or equal to a preset distance threshold, then it can be determined that there is no abrupt change in the depth information of the target obstacle in the depth image.

[0067] For example, the second threshold could be 15%, meaning the target device could identify objects with a reflectance higher than or equal to 5% and lower than 15% as low-reflectivity objects. In practical applications, low-reflectivity objects can include, but are not limited to, black carpets.

[0068] In step c, if the reflectivity of the target obstacle determined from the reflectivity image is higher than or equal to the second threshold and lower than the third threshold, the object type is determined to be a medium reflectivity object, and the subtype of the medium reflectivity object is determined based on the texture image.

[0069] It should be noted that when the reflectivity of the target obstacle is higher than or equal to the second threshold, it is not necessary to determine the object type of the target obstacle based on the depth image. This is because when the reflectivity of the target obstacle is higher than or equal to the second threshold, there is no significant difference in the depth images of different types of obstacles.

[0070] For example, the third threshold can be 50%, meaning the target device can classify objects with a reflectivity higher than or equal to 15% and lower than 50% as medium reflectivity objects.

[0071] If the target obstacle is determined to be a medium reflectivity object, its subtype can be determined based on the texture image acquired by the imaging device. In practical applications, subtypes of medium reflectivity objects can include, but are not limited to, basketballs, wooden benches, clothing, and vases.

[0072] It should be noted that if the target obstacle is determined to be an object type other than medium reflectivity (such as transparent, low reflectivity, and high reflectivity objects), then it is not necessary to determine the subtype of other object types based on the texture image acquired by the imaging device. This is because in practical applications, there are many subtypes of medium reflectivity objects, and the optimal control strategies for different subtypes of medium reflectivity objects are usually different. For example, the optimal control strategy for a target obstacle subtype of a basketball is different from the optimal control strategy for a target obstacle subtype of a vase. Based on this, after determining that the target obstacle is a medium reflectivity object, the subtype of medium reflectivity object can be determined based on the texture image acquired by the imaging device, thereby enabling better control of the target device to avoid obstacles in subsequent steps.

[0073] In practical applications, texture images can be input into a pre-trained deep learning model to indicate the subtype of the reflective object in the deep learning model output.

[0074] In step d, if the reflectivity of the target obstacle determined based on the reflectivity image is higher than or equal to the third threshold, then the object type is determined to be a high reflectivity object.

[0075] For example, the third threshold can be 50%, meaning the target device can identify objects with a reflectivity higher than or equal to 50% as high-reflectivity objects. In practical applications, high-reflectivity objects can include, but are not limited to, mirrors and metals.

[0076] In S104, the movement of the target device is controlled according to the object type.

[0077] In this embodiment of the application, after determining the object type of the target obstacle, the target device can also control the movement of the target device according to the object type of the target obstacle, so that the target device does not collide with the target obstacle during the movement.

[0078] In one possible implementation, the target device can control its movement based on the type of the target obstacle through steps e to g:

[0079] In step e, if the object type is any of the following: transparent object, low reflectivity object, or high reflectivity object, then the safe distance of the target obstacle is determined according to the object type. Different object types correspond to different safe distances.

[0080] Specifically, if the object type is transparent, the safe distance corresponding to the transparent object can be determined as the safe distance to the target obstacle; if the object type is low reflectivity, the safe distance corresponding to the low reflectivity object can be determined as the safe distance to the target obstacle; if the object type is high reflectivity, the safe distance corresponding to the high reflectivity object can be determined as the safe distance to the target obstacle.

[0081] In step f, if the object type is a medium reflectivity object, the safe distance of the target obstacle is determined according to the subtype. Different subtypes correspond to different safe distances.

[0082] For example, if the subtype of a medium reflectivity object is a vase, the safe distance corresponding to the vase can be determined as the safe distance to the target obstacle; if the subtype of a medium reflectivity object is a basketball, the safe distance corresponding to the basketball can be determined as the safe distance to the target obstacle.

[0083] In step g, the target device is controlled to move based on the safe distance and the current distance between the target device and the target obstacle.

[0084] In this implementation, after determining the safe distance, the target device can control its movement based on the safe distance and the current distance between the target device and the target obstacle.

[0085] In one possible implementation, the target device can determine the current distance between itself and the target obstacle based on the depth image.

[0086] However, when the object type is transparent, the depth information of the target obstacle in the depth image changes abruptly. Therefore, determining the current distance between the target device and the target obstacle with the object type of transparent object based on the depth image is inaccurate. Based on this, an ultrasonic distance sensor can be set in the target device in advance. After determining that the object type is transparent, the current distance between the target device and the transparent object can be determined based on the ultrasonic distance data obtained by the ultrasonic distance sensor.

[0087] Furthermore, when the object type is a low-reflectivity object, although there is no abrupt change in the depth information of the target obstacle in the depth image, the light reflectivity of the low-reflectivity object is relatively low. Therefore, determining the current distance between the target device and the target obstacle of the low-reflectivity object type based on the depth image is also inaccurate. Based on this, an ultrasonic distance sensor can be set in the target device in advance. After determining that the object type is a low-reflectivity object, the current distance between the target device and the low-reflectivity object can be determined based on the ultrasonic distance data obtained by the ultrasonic distance sensor.

[0088] Based on the above description, the target device can determine the current distance between itself and the target obstacle in the following ways:

[0089] If the target obstacle is a transparent or low-reflectivity object, the current distance between the target device and the target obstacle is determined based on the ultrasonic distance data obtained by the ultrasonic distance sensor.

[0090] If the target obstacle is a medium-reflectivity or high-reflectivity object, the current distance between the target device and the target obstacle is determined based on the depth image.

[0091] It should be noted that in practical applications, the specific method of controlling the movement of the target device can be set according to actual needs, based on the safety distance and the current distance between the target device and the target obstacle, and is not limited here.

[0092] In one possible implementation, during the process of controlling the movement of the target device according to the object type of the target obstacle, the target device may also perform step h:

[0093] In step h, based on the depth image, it is determined whether the target obstacle includes a hollow structure; if the target obstacle includes a hollow structure, the target device is controlled to move based on the structural information of the hollow structure and the object type.

[0094] For example, the openwork structure may include, but is not limited to, the space structure enclosed by the legs and tabletop of a table, and the space structure enclosed by the legs and seat of a stool.

[0095] In this implementation, the specific method for determining whether a target obstacle includes a hollow structure based on the depth image can be set according to actual needs.

[0096] If the target obstacle includes a hollow structure, the target device can control its movement by combining the structural information of the hollow structure and the object type; if the target obstacle does not include a hollow structure, the target device can control its movement based solely on the object type.

[0097] For example, the target device can be controlled to move by combining the structural information of the hollow structure and the object type as follows: The target device can first determine the bottom height of the hollow structure based on its structural information. If the bottom height of the hollow structure is less than a preset height threshold, the target device is controlled to move according to the safety distance corresponding to the object type, so that the target device does not collide with the target obstacle during movement. If the bottom height of the hollow structure is greater than or equal to the preset height threshold, the target device is controlled to move within the hollow structure. The preset height threshold can be set according to actual needs.

[0098] In another possible implementation, during the process of controlling the movement of the target device according to the object type of the target obstacle, the target device may also perform step i:

[0099] In step i, based on the depth image, it is determined whether the target obstacle includes a ramp structure; if the target obstacle includes a ramp structure, the target device is controlled to move based on the structural information of the ramp structure and the object type.

[0100] For example, ramp structures may include, but are not limited to, ramp stairs, slides, etc.

[0101] In this implementation, the specific method for determining whether a target obstacle includes a ramp structure based on the depth image can be set according to actual needs.

[0102] If the target obstacle includes a ramp structure, the target device can control its movement by combining the structural information of the ramp structure and the object type; if the target obstacle does not include a ramp structure, the target device can control its movement based solely on the object type.

[0103] For example, the target device can control its movement by combining the structural information of the ramp structure and the object type: First, the target device can determine the slope of the ramp structure based on the structural information. If the slope is greater than a preset ramp threshold, the target device's movement is controlled according to the safety distance corresponding to the object type, ensuring that the target device does not collide with the target obstacle during movement. If the slope is less than or equal to the preset ramp threshold, the target device is controlled to climb the ramp structure, allowing it to pass through. The preset ramp threshold can be set according to actual needs.

[0104] It should be noted that during the process of controlling the movement of the target device based on the object type of the target obstacle, steps e to g, h, and i can be executed simultaneously. That is, after determining the object type of the target obstacle, the target device can determine whether the target obstacle includes a hollow structure and / or a ramp structure. If the target obstacle includes a hollow structure and / or a ramp structure, the target device can control the movement of the target device based on the structural information of the hollow structure and / or the ramp structure, the safety distance corresponding to the object type, and the current distance between the target device and the target obstacle.

[0105] It can be seen that by fully acquiring comprehensive information about the target obstacle (such as the structural information of the hollow structure, the structural information of the ramp structure, the safe distance corresponding to the object type, and the current distance between the target device and the target obstacle) before controlling the movement of the target device, the obstacle avoidance accuracy of the target device is improved.

[0106] Furthermore, given that the structural information of the hollow structure meets certain conditions, the target device can be controlled to move within the hollow structure. When the target device is a robotic vacuum cleaner, this allows the robot to clean the hollow structure within the target obstacle, thus increasing its coverage area. Similarly, given that the structural information of the ramp structure meets certain conditions, the target device can also be controlled to move on the ramp structure. When the target device is a robotic vacuum cleaner, this allows it to clean the ramp structure within the target obstacle, further increasing its coverage area.

[0107] In practical applications, the specific implementation method for controlling the movement of the target device can be set according to actual needs, based on the structural information of the hollow structure, the structural information of the ramp structure, the safety distance corresponding to the object type, and the current distance between the target device and the target obstacle. This is not limited here.

[0108] As can be seen from the above, the device control method provided in this application is applied to a target device equipped with a three-dimensional distance sensor and an imaging device. First, the three-dimensional distance sensor acquires depth and reflectance images of the environment in which the target device is located, and the imaging device acquires texture images of the environment in which the target device is located. Then, based on the depth, reflectance, and texture images, the object type of the target obstacle in the environment in which the target device is located is determined. Finally, based on the object type, the movement of the target device is controlled. The device control method of this application can fully acquire omnidirectional information about the target obstacle based on the depth, reflectance, and texture images of the environment in which the target device is located before controlling the movement of the target device. Compared with the prior art, the device control method of this application can acquire not only the spatial information of the target obstacle but also the information of target obstacles with low reflectance before controlling the movement of the target device. Therefore, the device control method of this application can improve the obstacle avoidance accuracy of the target device.

[0109] In one possible implementation, the target device may also perform the following steps j to k, as detailed below:

[0110] In step j, the environmental complexity of the environment in which the target device is located is determined based on the depth image, reflectance image, and texture image.

[0111] In this implementation, the target device can pre-build an environmental complexity assessment model. After acquiring the depth image, reflectivity image, and texture image, the target device can input the depth image, reflectivity image, and texture image into the environmental complexity assessment model to instruct the environmental complexity assessment model to output the environmental complexity of the environment in which the target device is located.

[0112] For example, the environmental complexity assessment model can determine the total number of obstacles in the environment in which the target device is located, the number of dynamic obstacles in the environment in which the target device is located, the number of hollow structures and the number of ramp structures of all target obstacles in the environment in which the target device is located, based on depth images, reflectivity images, and texture images. Then, based on the total number of obstacles in the environment in which the target device is located, the number of dynamic obstacles in the environment in which the target device is located, the number of hollow structures and the number of ramp structures of all target obstacles in the environment in which the target device is located, the environmental complexity is determined.

[0113] The greater the total number of obstacles in the environment where the target device is located, the higher the environmental complexity can be. The greater the number of dynamic obstacles in the environment where the target device is located, the higher the environmental complexity can be. The greater the number of hollow structures and ramp structures of all target obstacles in the environment where the target device is located, the higher the environmental complexity can be.

[0114] In step k, the operating parameters of the three-dimensional distance sensor are adjusted according to the environmental complexity; the operating parameters include image resolution, and the relationship between environmental complexity and image resolution is positively correlated.

[0115] In this implementation, the operating parameters may include at least the image resolution, and may also include the image acquisition frequency, etc.

[0116] For example, environmental complexity can be divided into 0 to 10 points. When the environmental complexity is 0 to 2 points, the target device can be considered to be in an open space, and the target device can adjust the image resolution of the 3D distance sensor to 80×60. When the environmental complexity is 2 to 5 points, the target device can be considered to be in a simple space, and the target device can adjust the image resolution of the 3D distance sensor to 160×120. When the environmental complexity is 5 to 8 points, the target device can be considered to be in a dynamic obstacle area, and the target device can adjust the image resolution of the 3D distance sensor to 320×240. When the environmental complexity is 8 to 10 points, the target device can be considered to be in a high-density obstacle area or a steep slope area, and the target device can adjust the image resolution of the 3D distance sensor to 640×480.

[0117] It can be seen that by going through steps j to k, the working parameters of the three-dimensional distance sensor can be adjusted according to the environmental complexity of the target device's environment. This can solve the problem of wasted computing power of the target device, reduce the energy consumption of the target device, and accelerate the response efficiency of the target device when the environmental complexity of the target device's environment is low.

[0118] In one possible implementation, the target device may also perform the following steps 1 to n, as detailed below:

[0119] In step 1, the location and structural information of the target obstacle are determined based on the depth image, reflectance image, and texture image; the structural information includes any one or more of the height, width, and length information.

[0120] In this implementation, the target device can determine the position and structure information of the target obstacle based on the depth image, reflectivity image, and texture image by using a pre-trained position information determination model and structure information determination model.

[0121] The location information of the target obstacle can be a coordinate matrix composed of several coordinate points, and each coordinate point can be determined according to the preset coordinate axis.

[0122] Optionally, the structural information may also include the hollow structure information of the hollow structure in the target obstacle, and / or the ramp structure information of the ramp structure in the target obstacle.

[0123] In step m, if the object type is any of the following: transparent object, low reflectivity object, or high reflectivity object, the target obstacle is marked in the preset electronic map according to the object type, location information, and structural information.

[0124] In this implementation, the preset electronic map can be an electronic map corresponding to the environment in which the target device is located. The preset electronic map can be set on the terminal device used by the user.

[0125] For example, if the object type is any of the following: transparent object, low reflectivity object, or high reflectivity object, such as glass (transparent object), black carpet (low reflectivity object), or mirror (high reflectivity object), the target device can mark the object type, location information, and structural information of the glass, black carpet, or mirror in a preset electronic map. This allows the user to obtain the object type, location information, and structural information of the target obstacle through the preset electronic map, thereby optimizing the control strategy of the target device and further improving the obstacle avoidance accuracy of the target device.

[0126] In step n, if the object type is a medium reflectivity object, the target obstacle is marked in the preset electronic map according to the subtype, location information and structural information.

[0127] In this implementation, if the target obstacle is a medium reflectivity object, such as a basketball, a wooden stool, clothes, or a vase, the target device can mark the subtype, location information, and structural information of the basketball, wooden stool, clothes, or vase on a preset electronic map. This allows the user to obtain the subtype, location information, and structural information of the target obstacle through the preset electronic map, thereby optimizing the control strategy of the target device and further improving the obstacle avoidance accuracy of the target device.

[0128] Based on the device control method provided in the above embodiments, this application further provides a device control apparatus for implementing the above method embodiments. This device control apparatus can be applied to a target device equipped with a three-dimensional distance sensor and a shooting device. Please refer to... Figure 2 , Figure 2 This is a schematic diagram of the structure of a device control apparatus provided in an embodiment of this application. Figure 2 As shown, the device control unit 20 may include: a first image acquisition unit 21, a second image acquisition unit 22, an object type determination unit 23, and a control unit 24. Wherein:

[0129] The first image acquisition unit 21 is used to acquire depth images and reflectance images of the environment in which the target device is located through a three-dimensional distance sensor.

[0130] The second image acquisition unit 22 is used to acquire texture images of the environment in which the target device is located through the shooting device.

[0131] The object type determination unit 23 is used to determine the object type of the target obstacle in the environment where the target device is located based on the depth image, reflectance image and texture image.

[0132] The control unit 24 is used to control the movement of the target device according to the type of object.

[0133] Optionally, the object type determination unit 23 is specifically used for:

[0134] If the depth information of the target obstacle in the depth image changes abruptly, and the reflectivity of the target obstacle determined from the reflectivity image is lower than the first threshold, then the object type is determined to be a transparent object.

[0135] If the depth information of the target obstacle in the depth image does not change abruptly, and the reflectivity of the target obstacle determined from the reflectivity image is higher than or equal to the first threshold and lower than the second threshold, then the object type is determined to be a low reflectivity object.

[0136] If the reflectance of the target obstacle determined based on the reflectance image is higher than or equal to the second threshold and lower than the third threshold, then the object type is determined to be a medium reflectance object, and the subtype of the medium reflectance object is determined based on the texture image.

[0137] If the reflectivity of the target obstacle determined based on the reflectivity image is higher than or equal to the third threshold, then the object type is determined to be a high-reflectivity object.

[0138] Optionally, the control unit 24 is specifically used for:

[0139] If the object type is any of the following: transparent, low reflectivity, or high reflectivity, then the safe distance to the target obstacle is determined according to the object type, with different object types corresponding to different safe distances;

[0140] If the object type is a medium reflectivity object, then the safe distance to the target obstacle is determined according to the subtype, and different subtypes correspond to different safe distances;

[0141] The movement of the target device is controlled based on the safe distance and the current distance between the target device and the target obstacle.

[0142] Optionally, the control unit 24 is also used for:

[0143] Based on the depth image, the reflectivity image, and the texture image, the location and structural information of the target obstacle are determined; the structural information includes any one or more of height, width, and length information.

[0144] If the object type is any of the transparent object, the low reflectivity object, or the high reflectivity object, then the target obstacle is marked in a preset electronic map according to the object type, the location information, and the structural information;

[0145] If the object type is a medium reflectivity object, then the target obstacle is marked in a preset electronic map according to the subtype, the location information, and the structural information.

[0146] Optionally, the control unit 24 is also used for:

[0147] Based on the depth image, determine whether the target obstacle includes a hollow structure;

[0148] When the target obstacle includes a hollow structure, the movement of the target device is controlled based on the structural information of the hollow structure and the object type.

[0149] Optionally, the control unit 24 is also used for:

[0150] Based on the depth image, determine whether the target obstacle includes a ramp structure;

[0151] When the target obstacle includes a ramp structure, the movement of the target device is controlled based on the structural information of the ramp structure and the type of object.

[0152] Optionally, the three-dimensional distance sensor and the imaging device are coaxially mounted on the target device; the device control unit 20 may also include an alignment processing unit, wherein:

[0153] The alignment processing unit is specifically used for:

[0154] Based on the timestamps corresponding to the depth image, reflectance image, and texture image, time alignment processing is performed on the depth image, reflectance image, and texture image;

[0155] Spatial alignment processing is performed on depth images, reflectivity images, and texture images based on the rotation matrix and translation vector between the 3D distance sensor and the imaging device.

[0156] Optionally, the equipment control device 20 may also include a parameter adjustment unit. Wherein:

[0157] The parameter adjustment unit is specifically used for:

[0158] The environmental complexity of the target device's environment is determined based on depth images, reflectance images, and texture images.

[0159] The operating parameters of the 3D distance sensor are adjusted according to the environmental complexity; among these parameters is image resolution, and the relationship between environmental complexity and image resolution is positively correlated.

[0160] It should be noted that the information interaction and execution process between the above-mentioned units are based on the same concept as the method embodiments of this application. Their specific functions and technical effects can be referred to the method embodiments section, and will not be repeated here.

[0161] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a target device provided in an embodiment of this application. Figure 3 As shown, the target device 3 provided in this embodiment may include: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a program corresponding to the device control method. When the processor 30 executes the computer program 32, it implements the steps described above in the embodiment of the device control method, for example... Figure 1 S101 to S104 are shown. Alternatively, when the processor 30 executes the computer program 32, it implements the functions of each module / unit in the above-described device control device embodiment, for example... Figure 2 The functions of units 21 to 24 shown.

[0162] For example, computer program 32 can be divided into one or more modules / units, one or more of which are stored in memory 31 and executed by processor 30 to complete this application. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of computer program 32 in target device 3. For example, computer program 32 can be divided into a first image acquisition unit 21, a second image acquisition unit 22, an object type determination unit 23, and a control unit 24. For the specific functions of each unit, please refer to... Figure 2 The relevant descriptions in the corresponding embodiments are not repeated here.

[0163] Those skilled in the art will understand that Figure 3 This is merely an example of target device 3 and does not constitute a limitation on target device 3. It may include more or fewer components than shown, or combine certain components, or use different components.

[0164] The processor 30 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0165] The memory 31 can be an internal storage unit of the target device 3, such as a hard disk or RAM of the target device 3. The memory 31 can also be an external storage device of the target device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, or flash card equipped on the target device 3. Furthermore, the memory 31 can include both internal and external storage units of the target device 3. The memory 31 is used to store computer programs and other programs and data required by the target device. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0166] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units is merely an example. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the device control unit can be divided into different functional units to complete all or part of the functions described above. The functional units in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0167] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the various method embodiments described above.

[0168] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0169] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, refer to the relevant descriptions of other embodiments.

[0170] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0171] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A device control method, characterized in that, The method, applied to a target device equipped with a three-dimensional distance sensor and an imaging device, includes: The three-dimensional distance sensor is used to acquire depth and reflectance images of the environment in which the target device is located. The imaging device acquires a texture image of the environment in which the target device is located. Based on the depth image, the reflectivity image, and the texture image, determine the object type of the target obstacle in the environment where the target device is located; The movement of the target device is controlled according to the type of object.

2. The method according to claim 1, characterized in that, The step of determining the object type of the target obstacle in the environment where the target device is located based on the depth image, the reflectivity image, and the texture image includes: If the depth information of the target obstacle in the depth image changes abruptly, and the reflectivity of the target obstacle determined according to the reflectivity image is lower than a first threshold, then the object type is determined to be a transparent object. If the depth information of the target obstacle in the depth image does not change abruptly, and the reflectivity of the target obstacle determined according to the reflectivity image is higher than or equal to the first threshold and lower than the second threshold, then the object type is determined to be a low reflectivity object. If the reflectance of the target obstacle determined based on the reflectance image is higher than or equal to the second threshold and lower than the third threshold, then the object type is determined to be a medium reflectance object, and the subtype of the medium reflectance object is determined based on the texture image. If the reflectivity of the target obstacle determined based on the reflectivity image is higher than or equal to the third threshold, then the object type is determined to be a high-reflectivity object.

3. The method according to claim 2, characterized in that, The step of controlling the movement of the target device according to the object type includes: If the object type is any of the transparent object, the low reflectivity object, or the high reflectivity object, then the safe distance of the target obstacle is determined according to the object type, and different object types correspond to different safe distances; If the object type is a medium reflectivity object, then the safe distance of the target obstacle is determined according to the subtype, and different subtypes correspond to different safe distances; The target device is controlled to move based on the safe distance and the current distance between the target device and the target obstacle.

4. The method according to claim 2, characterized in that, Also includes: The location and structural information of the target obstacle are determined based on the depth image, the reflectivity image, and the texture image. The structural information includes any one or more of the following: height information, width information, and length information; If the object type is any of the transparent object, the low reflectivity object, or the high reflectivity object, then the target obstacle is marked in a preset electronic map according to the object type, the location information, and the structural information; If the object type is a medium reflectivity object, then the target obstacle is marked in a preset electronic map according to the subtype, the location information, and the structural information.

5. The method according to claim 1, characterized in that, The step of controlling the movement of the target device according to the object type includes: Based on the depth image, determine whether the target obstacle includes a hollow structure; When the target obstacle includes the hollow structure, the movement of the target device is controlled according to the structural information of the hollow structure and the type of the object.

6. The method according to claim 1, characterized in that, The step of controlling the movement of the target device according to the object type includes: Based on the depth image, determine whether the target obstacle includes a ramp structure; When the target obstacle includes the ramp structure, the movement of the target device is controlled according to the structural information of the ramp structure and the type of the object.

7. The method according to claim 1, characterized in that, The three-dimensional distance sensor and the imaging device are coaxially mounted on the target device; Before determining the object type of the target obstacle in the environment where the target device is located based on the depth image, the reflectance image, and the texture image, the method further includes: Based on the timestamps corresponding to the depth image, reflectance image, and texture image, time alignment processing is performed on the depth image, reflectance image, and texture image; Spatial alignment processing is performed on the depth image, the reflectivity image, and the texture image based on the rotation matrix and translation vector between the three-dimensional distance sensor and the imaging device.

8. The method according to any one of claims 1 to 7, characterized in that, Also includes: The environmental complexity of the environment in which the target device is located is determined based on the depth image, the reflectivity image, and the texture image. The operating parameters of the three-dimensional distance sensor are adjusted according to the environmental complexity; wherein the operating parameters include image resolution, and the relationship between environmental complexity and image resolution is positively correlated.

9. A target device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements each step of the device control method as described in any one of claims 1 to 8.

10. A computer program product, characterized in that, When the computer program product is executed by a processor, it implements the steps of the device control method as described in any one of claims 1 to 8.

Images