A task object processing method and system, a camera device, and a mobile robot

Abstract

This application provides a task object processing method, system, camera device, and mobile robot, applied to robot control technology. The method, applied to a camera device, includes: acquiring image data of a target scene as first image data; detecting whether the first image data contains a target task object; the target task object being a task object of a mobile robot deployed within the target scene; when the target task object is detected from the first image data, determining the target task object's location information in the target scene, and sending a task execution request carrying the location information to the mobile robot, so that the mobile robot moves to the target task object according to the location information to execute the target task. This solution allows for timely processing of task objects within the target scene.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a task object processing method, system, camera equipment, and mobile robot. Background Technology

[0002] Mobile robots refer to robots that can move and are used to perform a specific task, such as robotic vacuum cleaners and housekeeping robots.

[0003] In related technologies, the time and location for mobile robots to perform tasks are often specified by the user. For example, a user might set a robot vacuum to clean the entire area or a designated area at 8:00 AM daily, and the robot will then clean the entire area or the designated area at 8:00 AM every day. However, outside of the user-specified time period, if trash accumulates in the area, the robot vacuum can only wait until the next cleaning cycle to clean it.

[0004] This shows that mobile robots in related technologies cannot handle unexpected tasks in a timely manner in business scenarios. Summary of the Invention

[0005] The purpose of this application is to provide a task object processing method, system, camera device, and mobile robot to process task objects existing in a target scene in a timely manner. The specific technical solution is as follows:

[0006] In a first aspect, embodiments of this application provide a task object processing method, applied to a camera device deployed in a target scene, the method comprising:

[0007] Image data of the target scene is collected as the first image data;

[0008] Detect whether the first image data contains a target task object; the target task object is the task object of the mobile robot deployed in the target scene;

[0009] When the target task object is detected from the first image data, the location information of the target task object in the target scene is determined, and a task execution request carrying the location information is sent to the mobile robot so that the mobile robot moves to the target task object according to the location information to execute the target task.

[0010] Optionally, determining the location information of the target task object in the target scene includes:

[0011] Determine the first position of the target task object in the first image data;

[0012] Based on the first mapping relationship between the pixel coordinate system corresponding to the camera device and the world coordinate system where the mobile robot is located, the second position mapped by the first position in the world coordinate system is determined as the position information of the target task object in the target scene.

[0013] Optionally, after determining the first position of the target task object in the first image data, the method further includes:

[0014] Determine whether the first location is located within the target area corresponding to the mobile robot; the target area corresponding to the mobile robot is: the area in the first image data that corresponds to the task area of ​​the mobile robot in the target scene;

[0015] If the first position is located within the target area, then the step of determining the second position mapped in the world coordinate system based on the first mapping relationship between the pre-established pixel coordinate system corresponding to the camera device and the world coordinate system where the mobile robot is located is executed.

[0016] Optionally, the target area corresponding to the mobile robot may be determined in the following ways:

[0017] The image data collected by the camera device for the target scene is used as the second image data;

[0018] Semantic segmentation is performed on the second image data to obtain the semantic categories of different image regions in the second image data;

[0019] From each image region of the second image data, an image region with a specified semantic category is determined as the target region corresponding to the mobile robot; wherein, the specified semantic category is a preset semantic category of the task execution region of the mobile robot.

[0020] Optionally, after determining the image region with a specified semantic category from each image region of the second image data as the target region corresponding to the mobile robot, the method further includes:

[0021] During the inspection of the task execution area by the mobile robot, the camera device is used to detect the mobile robot and / or the task execution end of the mobile robot in real time, so as to determine the image area in the third image data collected by the camera device where the mobile robot and / or the task execution end of the mobile robot is located, which is used as the detection area.

[0022] Based on the detection area, the target area corresponding to the mobile robot is adjusted.

[0023] Optionally, adjusting the target area corresponding to the mobile robot based on the detection area includes:

[0024] The detection area is incorporated into the target area corresponding to the mobile robot to obtain the adjusted target area; and / or,

[0025] According to a preset pixel expansion rule, the pixels contained in the detection area are expanded; the expanded detection area is then incorporated into the target area corresponding to the mobile robot to obtain the adjusted target area.

[0026] Optionally, the camera parameters of the camera device are adjustable, and the camera parameters include at least one of rotation parameters, movement parameters, and scaling parameters; the pixel coordinate system corresponding to the camera device is the pixel coordinate system of the fourth image data, and the fourth image data is the image data collected by the camera device when it is in the preset camera parameters;

[0027] The step of determining the second position mapped in the world coordinate system of the first position based on the pre-established first mapping relationship between the pixel coordinate system corresponding to the camera device and the world coordinate system of the mobile robot includes:

[0028] Feature point matching is performed on the first image data and the fourth image data to obtain the successfully matched feature points in the first image data and the fourth image data, which are used as target feature points;

[0029] Determine the third position of the target feature point in the first image data and the fourth position in the fourth image data;

[0030] Based on the third position and the fourth position, a second mapping relationship is determined between the pixel coordinate system of the first image data and the pixel coordinate system of the fourth image data;

[0031] According to the second mapping relationship, the position of the first position mapped in the pixel coordinate system of the fourth image data is determined as the fifth position;

[0032] Based on the first mapping relationship between the pixel coordinate system of the fourth image data and the world coordinate system where the mobile robot is located, the position of the fifth position mapped in the world coordinate system is determined, which is then used as the second position mapped by the first position in the world coordinate system.

[0033] Optionally, a first mapping relationship between the pixel coordinate system corresponding to the camera device and the world coordinate system where the mobile robot is located is established in the following manner, including:

[0034] The camera device acquires multiple fifth image data points of the target scene; each fifth image data point includes the mobile robot and / or the task execution terminal of the mobile robot; the task execution terminal is a component of the mobile robot that processes the task object.

[0035] For each fifth image data, determine the position of the mobile robot and / or the task execution end in the fifth image data as the sixth position corresponding to the fifth image data, and send a position acquisition request carrying the acquisition time of the fifth image data to the mobile robot to determine the position of the mobile robot and / or the task execution end in the world coordinate system at the acquisition time as the seventh position corresponding to the fifth image data;

[0036] Based on the sixth and seventh positions corresponding to each fifth image data, a first mapping relationship between the pixel coordinate system corresponding to the camera device and the world coordinate system is determined.

[0037] Optionally, the mobile robot has multiple task processing modes; wherein each task processing mode is used to process task objects of at least one object type.

[0038] Before sending the task execution request carrying the location information to the mobile robot, the method further includes:

[0039] Determine the object type of the target task object, and use it as the target object type;

[0040] Sending a task execution request carrying the location information to the mobile robot includes:

[0041] A task execution request carrying the location information and the type of the target object is sent to the mobile robot, so that the mobile robot processes the task object at the location information according to the task processing mode corresponding to the type of the target object.

[0042] Optionally, there may be multiple mobile robots;

[0043] Sending a task execution request to the mobile robot for the target task object includes:

[0044] From a plurality of mobile robots, a mobile robot capable of processing the target task object is selected as the target mobile robot; wherein, the mobile robot capable of processing the target task object includes: a mobile robot whose object type includes the object type of the task object, and / or whose task area in the target scene includes the location of the target task object.

[0045] The target mobile robot is controlled to process the target task object.

[0046] Optionally, the method further includes:

[0047] When no task object of the mobile robot is detected from the first image data, or when the mobile robot has completed the processing of the target task object, the step of collecting image data of the target scene as the first image data is returned.

[0048] Secondly, embodiments of this application provide a task object processing method applied to a mobile robot, the method comprising:

[0049] The system receives a task execution request carrying location information from a camera device deployed in a target scene; wherein the target scene is the scene in which the mobile robot is deployed; the location information is the location information in the target scene determined by the camera device for the target task object in the first image data; the first image data is image data collected by the camera device for the target scene; and the target task object is the task object of the mobile robot detected in the first image data.

[0050] Move to the target task object according to the location information to execute the target task.

[0051] Optionally, moving to the target task object according to the location information to execute the target task includes:

[0052] Based on the location information, a path is planned to obtain the path to be moved.

[0053] Move according to the path to be moved, and process the task object at the location information after the move is completed.

[0054] Optionally, the mobile robot has multiple task processing modes; wherein each task processing mode is used to process a task object of at least one object type; the task execution request also carries a target object type, which is the object type of the target task object determined by the camera device;

[0055] The processing of the task object at the location information after the movement ends includes:

[0056] After the movement is completed, the task object at the location information is processed according to the task processing mode corresponding to the target object type.

[0057] Optionally, the step of performing path planning based on the location information to obtain the path to be moved includes:

[0058] If the task execution request was received when the mobile robot was in standby mode, then a movement path from the current location of the mobile robot to the location information is determined as the path to be moved.

[0059] If the task execution request is received by the mobile robot while it is in a task state, then based on the task state and the location information, the mobile robot's movement path is planned as the path to be moved.

[0060] Optionally, the task state includes at least one of a global task state and a sudden task execution state; wherein, the global task state is the state in which the mobile robot is performing global task processing, and the sudden task state is the state in which the mobile robot is performing a task sent by the camera device.

[0061] If the task execution request is received by the mobile robot while it is in a task state, then based on the task state and the location information, a movement path for the mobile robot is planned as the path to be moved, including:

[0062] If the task execution request is received when the mobile robot is in a global task state, then it is determined whether the location information is within the area of ​​the executed task; if it is, then a movement path from the current location of the mobile robot to the location information is immediately determined as the path to be moved, or, after the global task execution ends, a movement path from the location at the end of the global task to the location information is determined as the path to be moved; otherwise, the movement path of the global task is taken as the path to be moved.

[0063] If the task execution request is received when the mobile robot is in a sudden task state, then when the mobile robot moves to the task object corresponding to the sudden task state, a movement path from the current position of the mobile robot to the position of the task object corresponding to the sudden task state and the location information are determined as the movement path to be moved; during the process of the mobile robot processing the task object corresponding to the sudden task state, a movement path from the current position of the mobile robot to the location information is determined as the movement path to be moved.

[0064] Optionally, the step of performing path planning based on the location information to obtain the path to be moved includes:

[0065] Write the location information into the location queue;

[0066] Based on at least one position contained in the position queue, the mobile robot's movement path is planned as the path to be moved.

[0067] Thirdly, embodiments of this application provide a task object processing system, the task object processing system comprising: a camera device and a mobile robot;

[0068] The camera device is deployed in the target scene to collect image data of the target scene as first image data; detect whether the first image data contains a target task object; the target task object is the task object of the mobile robot deployed in the target scene; when the target task object is detected from the first image data, determine the location information of the target task object in the target scene, and send a task execution request carrying the location information to the mobile robot;

[0069] The mobile robot is used to receive the task execution request sent by the camera device, and move to the target task object according to the location information to execute the target task.

[0070] Fourthly, embodiments of this application provide a camera device, including a camera, a processor, and a machine-readable storage medium;

[0071] The camera is used to acquire image data of the target scene;

[0072] The machine-readable storage medium stores machine-executable instructions that can be executed by the processor, which is prompted by the machine-executable instructions to implement any of the methods described in the first aspect.

[0073] Fifthly, embodiments of this application provide a mobile robot, including a processor and a machine-readable storage medium; the machine-readable storage medium stores machine-executable instructions that can be executed by the processor, the processor being prompted by the machine-executable instructions to implement any of the methods described in the second aspect.

[0074] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in either the first or second aspect.

[0075] Beneficial effects of the embodiments in this application:

[0076] This application provides a task object processing method, system, camera device, and mobile robot. The camera device can collect image data of a target scene as first image data, and then detect whether the first image data contains the task object of the mobile robot deployed in the target scene. When the target task object is detected from the first image data, the location information of the target task object in the target scene is determined, and a task execution request carrying the location information is sent to the mobile robot, so that the mobile robot can move to the target task object according to the location information to execute the target task. Since the camera device detects the target task object from the first image data, it indicates that the task object of the mobile robot exists in the target scene. At this time, the camera device sends a task execution request with the location information of the target task object in the target scene to the mobile robot. After receiving the task execution request, the mobile robot can move to the target task object according to the location information and execute the target task, thereby enabling the mobile robot to process the task object in the target scene in a timely manner.

[0077] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0078] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0079] Figure 1 A flowchart illustrating the first task object processing method applied to a camera device provided in this application embodiment;

[0080] Figure 2 A schematic diagram of a semantically segmented semantic image provided in an embodiment of this application;

[0081] Figure 3 A flowchart illustrating a second task object processing method applied to a camera device, provided in an embodiment of this application;

[0082] Figure 4 This is a schematic diagram of a waste area and a cleanable area provided in an embodiment of this application;

[0083] Figure 5 This is a flowchart illustrating a task object processing method for mobile robots provided in an embodiment of this application.

[0084] Figure 6 This is a schematic diagram of the structure of the task object processing system provided in the embodiments of this application;

[0085] Figure 7 This is a structural diagram of a task object processing apparatus for a camera device provided in an embodiment of this application;

[0086] Figure 8 This is a structural diagram of a task object processing device for a mobile robot provided in an embodiment of this application;

[0087] Figure 9 This is a schematic diagram of the structure of the camera device provided in the embodiments of this application;

[0088] Figure 10 This is a schematic diagram of the structure of the mobile robot provided in an embodiment of this application. Detailed Implementation

[0089] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0090] In order to process task objects existing in the target scene in a timely manner, embodiments of this application provide a task object processing method, system, camera device, and mobile robot.

[0091] It should be noted that, in specific applications, the aforementioned camera device can be a smart camera with data processing capabilities, an infrared thermal camera, or other cameras with data processing capabilities. Alternatively, the aforementioned camera device can include a camera and a data processing unit, which can be deployed together with the camera as a whole or deployed independently of the camera. Furthermore, the task object processing method provided in this application embodiment can be implemented through software, hardware, or a combination of both.

[0092] In one embodiment, the mobile robot can be a mobile robot used to perform a specific task, such as a mobile cleaning robot, a housekeeping robot, a flying robot, a water surface robot, or a wall-climbing robot. The mobile cleaning robot includes a sweeping robot for cleaning floors; the housekeeping robot includes a mobile robot for tasks such as tidying up desktops, organizing clothes, and moving objects; the flying robot includes a robot for tasks such as aerial patrol; the water surface robot includes a robot for clearing horizontal obstacles and checking for suspicious objects on the water surface; and the wall-climbing robot includes a robot for cleaning walls and glass windows.

[0093] The task object processing method for camera equipment provided in this application embodiment may include:

[0094] Collect image data of the target scene as the first image data;

[0095] Detect whether the first image data contains the target task object; the target task object is the task object of the mobile robot deployed in the target scene;

[0096] When a target task object is detected from the first image data, its location information in the target scene is determined, and a task execution request carrying the location information is sent to the mobile robot, so that the mobile robot can move to the target task object according to the location information to execute the target task.

[0097] In the above-described scheme of this application embodiment, when the camera device detects the target task object from the first image data, it indicates that there is a task object of the mobile robot in the target scene. At this time, the camera device sends a task execution request to the mobile robot, which contains the location information of the target task object in the target scene. After receiving the task execution request, the mobile robot can move to the target task object according to the location information and execute the target task, thereby enabling the mobile robot to process the task object in the target scene in a timely manner.

[0098] The task object processing method for camera equipment provided in this application will now be described in detail with reference to the accompanying drawings.

[0099] like Figure 1 As shown, this application embodiment provides a task object processing method applied to a camera device, including steps S101-S103, wherein:

[0100] S101, Collect image data of the target scene as the first image data;

[0101] The target scenario refers to the scenario in which the mobile robot needs to perform a task. For example, if the mobile robot is a mobile cleaning robot or a housekeeping robot, the target scenario can be an indoor space that needs to be cleaned; if the mobile robot is a flying robot, the target scenario can be an airspace where the task needs to be performed; if the mobile robot is a water surface robot, the target scenario can be a body of water where the task needs to be performed; and if the mobile robot is a wall-climbing robot, the target scenario can be a wall or window where the task needs to be performed.

[0102] It should be emphasized that the target scenario mentioned above can include multiple types of mobile robots performing tasks. For example, if the target scenario is indoors, indoors includes the floor corresponding to a sweeping robot, the wall or window corresponding to a wall-climbing robot, or the desktop, bed, sofa, etc. corresponding to a housekeeping robot.

[0103] The aforementioned camera equipment can be pre-deployed within the target scene, enabling real-time image data acquisition of the target scene. When implementing the scheme of this application embodiment, the image data of the target scene acquired by the camera equipment can be used as the first image data.

[0104] S102, Detect whether the first image data contains the target task object; the target task object is the task object of the mobile robot deployed in the target scene;

[0105] In this step, the target task object refers to the task object of the mobile robot deployed within the target scene. Different types of mobile robots may have different target tasks. For example, the target task object of a mobile cleaning robot can be garbage such as fruit peels, rice grains, and spilled liquids. When the mobile robots deployed in the target scene include mobile cleaning robots, the target task object can include garbage such as fruit peels, rice grains, spilled liquids, and stains. The target task object of a housekeeping robot can be objects that need to be cleaned, such as clothes, paper, and bottles. When the mobile robots deployed in the target scene include housekeeping robots, the target task object can include objects that need to be cleaned, such as clothes. The target task object of a flying robot can be birds that need to be defended against or dispersed. When the mobile robots deployed in the target scene include flying robots, the target task object can include birds that need to be defended against or dispersed. The target task object of a surface robot can be obstacles or suspicious objects floating on the water. The target task object of a wall-climbing robot can be stains on walls or windows. When the mobile robots deployed in the target scene include surface robots, the target task object can include stains on walls or windows.

[0106] It should be noted that only one type of mobile robot can be deployed in the target scene, or multiple types of mobile robots can be deployed. For example, in an indoor scene, there are mobile cleaning robots and housekeeping robots deployed. In this case, the target task objects include the task objects of the mobile cleaning robots and the task objects of the housekeeping robots. That is, the target task objects include garbage such as fruit peels, rice grains, spilled liquids, stains, and clothes that need to be cleaned.

[0107] In order to process the task objects existing in the target scene in a timely manner, this application determines whether there are task objects that need to be processed in the target scene by detecting whether the first image data contains the target task objects.

[0108] Optionally, after acquiring the first image data, target task object recognition can be performed on the first image data. If the target task object is recognized from the first image data, it is determined that the first image data contains the target task object; otherwise, it is determined that the first image data does not contain the target task object.

[0109] The above-mentioned target object identification method can be implemented in a variety of ways, including at least one of the following methods:

[0110] In the first identification method, an object detection algorithm is used to detect whether the first image data contains the target task object. This object detection algorithm can be a task object detection model built using neural networks such as R-CNN or Fast R-CNN. After acquiring the first image data, the first image data can be input into the task object detection model to detect whether it contains the target task object. For example, if the target task object includes garbage such as fruit peels, rice grains, spilled liquid, and stains, a pre-trained garbage detection model can be used to detect the garbage and determine whether the first image data contains such garbage.

[0111] In the second recognition method, a semantic segmentation algorithm is used to determine whether the first image data contains the target object. Optionally, the first image data is analyzed and processed using a semantic segmentation algorithm to obtain a semantic image, in which each pixel contains the semantic category of that pixel. Optionally, the aforementioned semantic segmentation algorithm can be a semantic segmentation model built using a U-Net network or a Mask R-CNN network. Thus, after obtaining the first image data, the first image data can be input into the semantic segmentation model to obtain the semantic image corresponding to the first image data.

[0112] Let's take the example of a target task involving garbage such as fruit peels, rice grains, spilled liquid, and stains. A semantic segmentation algorithm for these objects is pre-trained. The first image data is input into this trained algorithm to obtain a semantic image. Each pixel in the semantic image has a semantic category. In this example, the semantic categories include garbage and non-garbage. Of course, semantic categories can be further subdivided as needed; for example, garbage can be subdivided into fruit peels, rice grains, etc., and non-garbage can be subdivided into sofas, tables, trash cans, etc. If the semantic image contains pixel regions where all elements are classified as garbage, then the first image data contains garbage such as fruit peels, rice grains, spilled liquid, and stains.

[0113] For example, such as Figure 2 As shown in the figure, this application provides a schematic diagram of a semantic image after semantic segmentation. Different textures in the figure represent different semantic categories. Figure 2 In the diagram, area A represents the wardrobe category, area B the clothing category, area C the bed category, area D the wall category, area E the trash category, area F the floor category, and area G the chair category. Pixels in each area have a unique identifier corresponding to the semantic category of their respective area.

[0114] S103, when the target task object is detected from the first image data, the location information of the target task object in the target scene is determined, and a task execution request carrying the location information is sent to the mobile robot so that the mobile robot moves to the target task object according to the location information to execute the target task.

[0115] When a target task object is detected from the first image data, it indicates that there is a task object that needs to be processed in the target scene. At this time, it is necessary to process the target task object in the target scene in a timely manner. Therefore, when a target task object is detected from the first image data, a task execution request for the target task object can be sent to the mobile robot so that the mobile robot can process the target task object after receiving the task execution request.

[0116] In one implementation, to enable the mobile robot to process task objects within the target scene in a timely manner, a task processing instruction can be sent to the mobile robot. This instruction instructs the mobile robot to perform global task processing. For example, taking a mobile cleaning robot deployed indoors as an example, after determining that there is trash that needs to be processed indoors, a task processing instruction can be sent to the mobile cleaning robot. Upon receiving the instruction, the mobile cleaning robot performs global cleaning of the entire indoor area. Since global cleaning can process the entire cleanable area, timely disposal of indoor trash can be achieved. While the aforementioned global task processing can promptly address task objects within the target scene, performing global task processing every time a task object is detected results in frequent global operations by the mobile robot, leading to resource waste.

[0117] To avoid frequent global operations by the mobile robot and reduce resource waste, one implementation method involves controlling the mobile robot to process only target task objects within the target scene. Optionally, the task execution request can carry the position information of the target task object in the world coordinate system, allowing the mobile robot to move to the location of the target task object after receiving the task execution request, and then process only the target task object. In short, to avoid frequent global operations by the mobile robot and reduce resource waste, it is possible to control the mobile robot to process only target task objects within the target scene. In this case, it is necessary to determine the position information of the target task object in the target scene. This can be done by first determining the position information of the target task object in the target scene, and then sending a task execution request carrying the position information to the mobile robot, so that the mobile robot can move to the target task object according to the position information to execute the target task. The specific implementation process will be described in detail in subsequent embodiments and will not be repeated here.

[0118] After determining the location information, a task execution request carrying that location information can be generated. This allows the mobile robot to directly process the target task object at the specified location after receiving the task execution request.

[0119] Optionally, the mobile robot described above has multiple task processing modes, each mode being used to process task objects of at least one object type. For example, a mobile cleaning robot may use different modes to process solid waste and liquid waste. To improve task execution efficiency, the object type of the target task object can be determined before sending a task execution request carrying location information to the mobile robot. This target object type is then used to send a task execution request carrying location information and the target object type to the mobile robot, enabling the mobile robot to process the task object at the location information according to the task processing mode corresponding to the target object type.

[0120] The object type of the target task object can be determined by object recognition algorithm and / or semantic segmentation algorithm. After the target object type is determined, a task execution request carrying location information and target object type can be sent to the mobile robot at the same time. After receiving the task execution request, the mobile robot can determine the task processing mode corresponding to the target object type according to the preset correspondence between object type and task processing mode, and then process the task object at the location information according to the task processing mode corresponding to the target object type.

[0121] When there are multiple mobile robots deployed in the target scene, that is, when there are multiple mobile robots, when it is necessary to process the task objects in the target scene, a target mobile robot can be selected from the multiple mobile robots. For example, a mobile robot can be randomly selected from the multiple mobile robots as the target robot.

[0122] Furthermore, considering the different locations of the mobile robots and the different types of tasks they can handle, a mobile robot capable of processing the target task object can be selected from among multiple mobile robots. In this case, the mobile robot capable of processing the target task object can be selected as the target mobile robot.

[0123] Optionally, in one implementation, multiple mobile robots can be filtered based on the object type of the target task object and / or the location of the target task object, and a mobile robot that processes the target task object can be selected as the target mobile robot. In one implementation, the mobile robot that can process the target task object includes: a mobile robot whose object type includes the object type of the task object, and / or whose task area in the target scene includes the location of the target task object.

[0124] Different types of mobile robots may handle different types of tasks. For example, in an indoor scenario involving both mobile cleaning robots and household robots, the target tasks may include various types of trash that the mobile robot can handle, as well as various items that need to be organized that the household robot can handle. In this case, if trash is identified from the first image data, the mobile robot capable of handling the target task is a mobile cleaning robot. Conversely, if items to be organized are identified from the first image data, the mobile robot capable of handling the target task is a household robot.

[0125] Furthermore, the locations of different mobile robots may differ. For example, in an indoor scenario involving both mobile cleaning robots and wall-climbing robots, the target task object includes various types of trash that the mobile robot can handle, while the wall-climbing robot can handle stains such as liquids that need cleaning. If the liquid needs cleaning, both the mobile cleaning robot and the wall-climbing robot can handle it. However, the mobile cleaning robot's task area is the indoor floor, while the wall-climbing robot's task area is the indoor wall or glass. Therefore, if the liquid is on the floor, the mobile robot capable of handling the target task object is the mobile cleaning robot; if the liquid is on the wall, the mobile robot capable of handling the target task object is the wall-climbing robot. Alternatively, multiple suitable robots can be selected by combining both the type and location of the target task object.

[0126] In one implementation, when no task object of the mobile robot is detected from the first image data, or when the mobile robot has completed the processing of the target task object, the step of collecting image data of the target scene as the first image data is returned.

[0127] In the above-described scheme of this application embodiment, when the camera device detects the target task object from the first image data, it indicates that there is a task object of the mobile robot in the target scene. At this time, the camera device sends a task execution request to the mobile robot, which contains the location information of the target task object in the target scene. After receiving the task execution request, the mobile robot can move to the target task object according to the location information and execute the target task, thereby enabling the mobile robot to process the task object in the target scene in a timely manner.

[0128] based on Figure 1 The illustrated embodiments, such as Figure 3 As shown, this embodiment of the invention also provides a task object processing method applied to a camera device. The step of determining the location information of the target task object in the target scene, as referred to by the camera device, may include S301–S302, wherein:

[0129] S301, Determine the first position of the target task object in the first image data;

[0130] To avoid frequent global operations by the mobile robot and reduce resource waste, the mobile robot can be controlled to process only target task objects within the target scene. In this case, it is necessary to determine the position information of the target task object in the target scene. Optionally, the first position of the target task object in the first image data can be determined first. This first position can be the pixel coordinate position of the target task object in the first image data. Optionally, if the target task object is determined using an object recognition algorithm, the above pixel coordinate position can be the pixel coordinates of the center pixel and / or corner pixel of the detection box corresponding to the target task object. If the target task object is determined using a semantic segmentation algorithm, the above pixel coordinate position can be the pixel coordinates of the geometric center pixel and / or edge pixel of the image region to which the target task object belongs.

[0131] S302, based on the first mapping relationship between the pixel coordinate system corresponding to the camera device and the world coordinate system where the mobile robot is located, determine the position mapped in the world coordinate system of the first position as the second position, as the position information of the target task object in the target scene;

[0132] Depending on the type of camera equipment, the pixel coordinate system corresponding to the camera equipment can be different.

[0133] In the first scenario, if the camera device has a fixed camera angle and shooting parameters, then the pixel coordinate system corresponding to the camera device is the pixel coordinate system of the aforementioned first image data. In this case, the position mapped in the world coordinate system of the first position can be determined directly based on the first mapping relationship between the pre-established pixel coordinate system of the first image data and the world coordinate system where the mobile robot is located, and this position is used as the second position.

[0134] In the second scenario, the camera device is one with adjustable camera parameters. These parameters include at least one of rotation, translation, and scaling parameters, meaning the camera device can rotate, pan horizontally and / or vertically, and / or zoom in or out to acquire image data. Since the camera parameters are not fixed, in this case, the pixel coordinate system corresponding to the camera device can be the pixel coordinate system of the fourth image data, which is the image data acquired by the camera device when it is operating at preset camera parameters. These preset camera parameters can be the camera parameters the camera device is in when it is in its initial state.

[0135] In this case, the first position of the target task object in the first image data can be mapped to its position in the fourth image data.

[0136] Optionally, the mapping relationship between the pixel coordinate systems of the first image data and the fourth image data can be determined first.

[0137] In one implementation, feature point matching can be performed between the first image data and the fourth image data to obtain the successfully matched feature points in the first and fourth image data, which are then used as target feature points. Optionally, feature point matching algorithms such as SIFT (Scale-Invariant Feature Transform) and SURF (Speeded UpRobust Features) can be used for feature point matching.

[0138] After successfully matching target feature points, the third position of each target feature point in the first image data and the fourth position in the fourth image data can be determined. Based on the third and fourth positions, a second mapping relationship between the pixel coordinate systems of the first and fourth image data is determined. Optionally, there can be multiple successfully matched target feature points, allowing the calculation of a transformation matrix between the two pixel coordinate systems. This transformation matrix maps points in the first image data to corresponding points in the fourth image data. The transformation matrix can be an affine matrix or a perspective matrix, etc. After determining the second mapping relationship, which is the mapping relationship between the pixel coordinate systems of the first and fourth image data, the position mapped by the first position in the pixel coordinate system of the fourth image data can be determined according to the second mapping relationship. This position is then designated as the fifth position, which is the position of the target task object in the pixel coordinate system of the fourth image data. Furthermore, according to the pre-established first mapping relationship between the pixel coordinate system of the fourth image data and the world coordinate system of the mobile robot, the position mapped by the fifth position in the world coordinate system can be determined as the position mapped by the first position in the world coordinate system.

[0139] Optionally, the first mapping relationship between the pixel coordinate system corresponding to the camera device and the world coordinate system where the mobile robot is located can be determined in various ways. For example, the first mapping relationship between the pixel coordinate system corresponding to the camera device and the world coordinate system where the mobile robot is located can be established by the following steps, including steps A1-A:

[0140] Step A1: Acquire multiple fifth image data captured by the camera device for the target scene; wherein each fifth image data contains a mobile robot and / or the task execution end of the mobile robot; the task execution end is the component of the mobile robot that processes the task object;

[0141] For mobile cleaning robots, the surface on which they operate (the ground) is the same as the surface on which they move. In this case, the acquired fifth image data must include the mobile cleaning robot. For housekeeping robots, the surface on which they operate (sofa, bed, tabletop, etc.) is different from the surface on which they move. In this case, the acquired fifth image data must include the task execution end of the housekeeping robot, such as its robotic arm.

[0142] Step A2: For each fifth image data, determine the position of the mobile robot and / or task execution end in the fifth image data, which is taken as the sixth position corresponding to the fifth image data;

[0143] After obtaining the fifth image data, the pixel coordinates of the mobile robot and / or task execution end within that fifth image data can be determined for each fifth image data, serving as the sixth position corresponding to that fifth image data. Optionally, the position of the mobile robot and / or task execution end within the fifth image data can be determined using object recognition algorithms and / or semantic segmentation algorithms.

[0144] Step A3: Send a position acquisition request carrying the acquisition time of the fifth image data to the mobile robot to determine the position of the mobile robot and / or the task execution end in the world coordinate system at the acquisition time, which is the seventh position corresponding to the fifth image data;

[0145] In this step, the camera device can communicate with the mobile robot in real time, allowing the mobile robot to send its own position information and / or the position information of its task execution segment to the camera device in real time. After acquiring each fifth image data, the camera device can filter the acquisition time of each fifth image data and the position of the mobile robot and / or task execution segment in the world coordinate system from the recorded positions of each fifth image data acquisition time, and use this as the seventh position corresponding to that fifth image data.

[0146] Step A4: Based on the sixth and seventh positions corresponding to each fifth image data, determine the first mapping relationship between the pixel coordinate system corresponding to the camera device and the world coordinate system.

[0147] After obtaining the sixth and seventh positions corresponding to the third and fifth image data, the first mapping relationship between the pixel coordinate system and the world coordinate system corresponding to the camera device can be determined based on the sixth and seventh positions corresponding to each fifth image data.

[0148] For example, taking a mobile cleaning robot as an example, let the moving ground where the mobile cleaning robot is located be z = 0. At this time, the mapping relationship to be calibrated can be expressed as:

[0149]

[0150] Where (u,v) represents the sixth position of the mobile robot in the fifth image data, s represents the scale factor, which is usually taken as 1, (x b ,y b ,z b Let h be the seventh position of the mobile robot in the world coordinate system. 11 ~h 32 This is the homography matrix to be calibrated, containing intrinsic and extrinsic parameters of the camera device.

[0151] After obtaining the sixth and seventh positions corresponding to each fifth image data, the least squares method can be used to substitute the sixth and seventh positions corresponding to each fifth image data into the above-mentioned mapping relationship to be calibrated, so as to determine the homography matrix containing the intrinsic and extrinsic parameters of the camera device to be calibrated, thereby obtaining the first mapping relationship between the pixel coordinate system and the world coordinate system corresponding to the camera device.

[0152] For example, in the case where the camera device is a camera and the mobile robot is a cleaning robot, continuing with the above example, when constructing the semantic image of the camera, the cleaning robot, during the inspection process, can obtain the cleaning robot's position information (x, y, z) in the world coordinate system. robot y robot The image is transmitted to the camera, and the camera simultaneously transmits the position (x, y) of the cleaning robot in the pixel coordinate system detected in the image. ipc y ipc To obtain multiple sets of corresponding position information (at least 4 sets, and the pixel coordinates in any 3 sets of position information are not collinear), since the cleaning robot moves on the ground, we can assume that z=0 in the world coordinate system. Then, through the multiple sets of corresponding position information of the two planes obtained in the previous step, we can obtain the first mapping relationship between the pixel coordinate system and the world coordinate system, namely the homography matrix H (3*3 matrix). Further, we can obtain the camera's extrinsic parameters (rotation R and translation T) and intrinsic parameters K.

[0153] In the above-described solution of this application embodiment, the mobile robot can process task objects existing in the target scene in a timely manner. Simultaneously, by sending a task execution request carrying the location information of the target task object to the mobile robot, frequent global operations by the mobile robot can be avoided, reducing resource waste.

[0154] In one embodiment, based on Figure 3 In the embodiment shown, after step S301, it can be determined whether the first position is located within the target area corresponding to the mobile robot. If the first position is located within the target area, then step S302 is executed. If the first position is not located within the target area, it means that the target task object does not need to be processed by the mobile robot, and the target task object can be ignored at this time.

[0155] The target area corresponding to the mobile robot is the area in the first image data that corresponds to the task area of ​​the mobile robot in the target scene. For example, for a robotic vacuum cleaner, its task area is the ground area, such as... Figure 2 The F region in the diagram; for a wall-climbing robot, its task area is the wall-climbing area, such as... Figure 2 The D area in the text; for domestic robots, their task area is the bed surface or tabletop area, such as... Figure 2 Region C in the text.

[0156] For example, in the case where the camera device is a video camera, the mobile robot is a cleaning robot, and the target task object is garbage, such as Figure 4 As shown in the illustration, this application provides a schematic diagram of trash and a cleanable area. For example, a camera performs real-time detection of the target scene, detecting whether there is trash to be cleaned in the first image data. A semantic segmentation algorithm is used to analyze and process the first image data to obtain a semantic image, and then it is detected whether the first location of the trash type in the semantic image is located in the target area corresponding to the cleaning robot (i.e.,...). Figure 4 Within the cleanable area (in the image). Optionally, it can be determined whether the pixel at the first position is within the pixel set of the target area corresponding to the cleaning robot. Optionally, the pixel at the first position is the semantic pixel set A to be cleaned in the semantic image (i.e., the pixel set of garbage in the scene), and then it can be determined that the semantic pixel set A and the pixel set B of the target area corresponding to the cleaning robot intersect to obtain the intersection C. If the intersection C is empty, it means that if the first position is not within the target area, it means that the target task object does not need to be processed by the robot, and the target task object can be ignored. If the intersection C is not empty, it means that the first position is within the target area, and step S302 can be executed.

[0157] The target area corresponding to the mobile robot can be predetermined or optional. Image data of the target scene can be collected as second image data, and then semantic segmentation can be performed on the second image data to obtain the semantic categories of different image regions in the second image data. From each image region of the second image data, the image region with the specified semantic category can be determined as the target area corresponding to the mobile robot. The specified semantic category is the semantic category of the preset task execution area of ​​the mobile robot.

[0158] by Figure 2 Taking the second image data as an example, after obtaining the second image data, semantic segmentation can be performed on the second image data to obtain, as shown below. Figure 2 The semantic image shown contains semantic categories for image regions. If the preset mobile robot includes a floor cleaning robot, then the semantic category is designated as the ground category, thereby determining region F from each image region as the image region with the specified semantic category, which serves as the target area for the floor cleaning robot.

[0159] In one implementation, after determining the image region with a specified semantic category from each image region of the second image data as the target region corresponding to the mobile robot, during the inspection of the mobile robot's task execution area, a camera device can be used to perform real-time detection on the mobile robot and / or the task execution end of the mobile robot to determine the image region where the mobile robot and / or the task execution end of the mobile robot is located in the third image data collected by the camera device, which is used as the detection region, and then the target region corresponding to the mobile robot is adjusted based on the detection region.

[0160] In one implementation, the detection area can be incorporated into the target area corresponding to the mobile robot to obtain an adjusted target area; and / or, the pixels contained in the detection area can be expanded according to a preset pixel expansion rule; the expanded detection area can be incorporated into the target area corresponding to the mobile robot to obtain an adjusted target area.

[0161] For example, when the mobile robot described above is a cleaning robot, the camera device can capture a scene image F, and then perform semantic segmentation on image F to generate a semantic image MAP. At this time, the cleaning robot performs a full-house cleaning area inspection, and the camera captures image data in real time to perform target detection on the cleaning robot, obtaining a series of cleaning robot detection boxes (w n h n ) and center point location information (x n y n ), where n is the number of detection boxes and location information, and thus the obtained (x n y n ) as the center, with min(w) n h n Using a radius of 1 / 2, a series of circular seed regions are formed in the image. These seed regions are then used for interactive semantic segmentation to fine-tune the cleanable areas of the ground in the semantic image MAP, resulting in the final semantic image MAP. seg-ipc .

[0162] In the above-described solution of this application embodiment, the mobile robot can process task objects existing in the target scene in a timely manner. Furthermore, by determining whether the first location is within the target area, unnecessary processing can be avoided.

[0163] Corresponding to the task object processing method for camera equipment provided in the above embodiments of this application, such as Figure 5 As shown, this application embodiment also provides a task object processing method applied to a mobile robot, including steps S501-S502, wherein:

[0164] S501, receive a task execution request carrying location information sent by a camera device deployed in the target scene; wherein, the target scene is the scene in which the mobile robot is deployed; the location information is the location information in the target scene determined by the camera device for the target task object in the first image data; the first image data is the image data collected by the camera device for the target scene; the target task object is the task object of the mobile robot detected in the first image data;

[0165] S502, Move to the target task object according to the location information to execute the target task.

[0166] In one implementation, the location information is the location information in the target scene determined by the camera device for the target task object in the first image data; the first image data is the image data collected by the camera device for the target scene; and the target task object is the task object of the mobile robot detected in the first image data.

[0167] In this case, path planning can be performed based on location information to obtain the path to be moved, and then the movement can be carried out according to the path to be moved. After the movement is completed, the task object at the location information is processed.

[0168] Optionally, in one implementation, the mobile robot has multiple task processing modes; each task processing mode is used to process a task object of at least one object type; the task execution request also carries the target object type, which is the object type of the target task object determined by the camera device.

[0169] In this case, after the movement is completed, the task object at the location information can be processed according to the task processing mode corresponding to the target object type.

[0170] In one implementation, the mobile robot has multiple states. If the task execution request is received when the mobile robot is in standby state, then a movement path from the current position of the mobile robot to the location information is determined as the path to be moved. If the task execution request is received when the mobile robot is in task state, then a movement path of the mobile robot is planned based on the task state and location information as the path to be moved.

[0171] Optionally, the above-mentioned task state includes at least one of global task state and sudden task execution state; wherein, the global task state is the state in which the mobile robot is when performing global task processing, and the sudden task state is the state in which the mobile robot is when performing the task sent by the camera device.

[0172] At this point, if the task execution request was received by the mobile robot while it was in a global task state, then it is determined whether the location information is within the area of ​​the executed task; if it is, then the movement path from the current location of the mobile robot to the location information is immediately determined as the path to be moved, or, after the global task execution is completed, the movement path from the location at the end of the global task to the location information is determined as the path to be moved; otherwise, the movement path of the global task is used as the path to be moved.

[0173] If the task execution request is received by the mobile robot when it is in a sudden task state, then when the mobile robot moves to the task object corresponding to the sudden task state, a movement path and its location information from the current position of the mobile robot to the position of the task object corresponding to the sudden task state are determined as the movement path to be moved; when the mobile robot is processing the task object corresponding to the sudden task state, a movement path and its location information from the current position of the mobile robot are determined as the movement path to be moved.

[0174] In one implementation, the path planning based on the location information is performed to obtain the path to be moved. At this time, the location information can be written into the location queue, and then the movement path of the mobile robot can be planned based on at least one location contained in the location queue, which is used as the path to be moved.

[0175] In the above-described scheme of this application embodiment, when the camera device detects the target task object from the first image data, it indicates that there is a task object of the mobile robot in the target scene. At this time, the camera device sends a task execution request to the mobile robot, which contains the location information of the target task object in the target scene. After receiving the task execution request, the mobile robot can move to the target task object according to the location information and execute the target task, thereby enabling the mobile robot to process the task object in the target scene in a timely manner.

[0176] like Figure 6 As shown, this application embodiment provides a task object processing system, which includes: a camera device 601 and a mobile robot 602;

[0177] Camera device 601 is deployed in the target scene to collect image data of the target scene as first image data; detect whether the first image data contains the task object of the mobile robot deployed in the target scene; when the target task object is detected from the first image data, determine the location information of the target task object in the target scene, and send a task execution request carrying the location information to the mobile robot.

[0178] Mobile robot 602 is used to receive task execution requests sent by camera equipment and move to the target task object according to the location information to execute the target task.

[0179] In the above-described scheme of this application embodiment, when the camera device detects the target task object from the first image data, it indicates that there is a task object of the mobile robot in the target scene. At this time, the camera device sends a task execution request to the mobile robot, which contains the location information of the target task object in the target scene. After receiving the task execution request, the mobile robot can move to the target task object according to the location information and execute the target task, thereby enabling the mobile robot to process the task object in the target scene in a timely manner.

[0180] Corresponding to the task object processing method for camera equipment provided in the above embodiments of this application, such as Figure 7 As shown in the illustration, this application also provides a task object processing device, applied to a camera device deployed in a target scene, the device comprising:

[0181] Data acquisition module 701 is used to collect image data of the target scene as the first image data;

[0182] The object detection module 702 is used to detect whether the first image data contains a target task object; the target task object is the task object of the mobile robot deployed in the target scene.

[0183] The request sending module 703 is used to determine the location information of the target task object in the target scene when the target task object is detected from the first image data, and send a task execution request carrying the location information to the mobile robot so that the mobile robot moves to the target task object according to the location information to execute the target task.

[0184] Optionally, the request sending module is specifically used to determine the first position of the target task object in the first image data; based on the first mapping relationship between the pixel coordinate system corresponding to the camera device and the world coordinate system where the mobile robot is located, determine the position mapped in the world coordinate system of the first position as the second position; and generate a task execution request carrying the second position.

[0185] Optionally, the request sending module is further configured to, after determining the first position of the target task object in the first image data, determine whether the first position is located within the target area corresponding to the mobile robot; the target area corresponding to the mobile robot is: the area in the first image data corresponding to the task area of ​​the mobile robot in the target scene; if the first position is located within the target area, then the step of determining the second position mapped in the world coordinate system based on the first mapping relationship between the pixel coordinate system corresponding to the camera device and the world coordinate system where the mobile robot is located is executed.

[0186] Optionally, the device further includes: a region determination module, configured to acquire image data collected by the camera device for the target scene as second image data; perform semantic segmentation on the second image data to obtain semantic categories of different image regions in the second image data; determine image regions with a specified semantic category from each image region of the second image data as target regions corresponding to the mobile robot; wherein the specified semantic category is a preset semantic category of the task execution region of the mobile robot.

[0187] Optionally, the region determination module is further configured to, after determining an image region with a specified semantic category from each image region of the second image data as the target region corresponding to the mobile robot, during the inspection of the task execution area of ​​the mobile robot, use the camera device to perform real-time detection on the mobile robot and / or the task execution end of the mobile robot to determine the image region where the mobile robot and / or the task execution end of the mobile robot is located in the third image data collected by the camera device, as the detection region; and adjust the target region corresponding to the mobile robot based on the detection region.

[0188] Optionally, the region determination module is specifically used to merge the detection region into the target region corresponding to the mobile robot to obtain an adjusted target region; and / or to expand the pixels contained in the detection region according to a preset pixel expansion rule; and to merge the expanded detection region into the target region corresponding to the mobile robot to obtain an adjusted target region.

[0189] Optionally, the camera parameters of the camera device are adjustable, and the camera parameters include at least one of rotation parameters, movement parameters, and scaling parameters; the pixel coordinate system corresponding to the camera device is the pixel coordinate system of the fourth image data, and the fourth image data is the image data collected by the camera device when it is in the preset camera parameters;

[0190] The request sending module is specifically configured to perform feature point matching on the first image data and the fourth image data to obtain successfully matched feature points in the first image data and the fourth image data as target feature points; determine the third position of the target feature point in the first image data and the fourth position in the fourth image data; based on the third position and the fourth position, determine a second mapping relationship between the pixel coordinate system of the first image data and the pixel coordinate system of the fourth image data; according to the second mapping relationship, determine the position mapped by the first position in the pixel coordinate system of the fourth image data as a fifth position; according to the pre-established first mapping relationship between the pixel coordinate system of the fourth image data and the world coordinate system where the mobile robot is located, determine the position mapped by the fifth position in the world coordinate system as the position mapped by the first position in the world coordinate system.

[0191] Optionally, the device further includes a mapping relationship establishment module, used to acquire multiple fifth image data captured by the camera device for the target scene; wherein each fifth image data includes the mobile robot and / or the task execution end of the mobile robot; the task execution end is a component of the mobile robot that processes the task object; for each fifth image data, the position of the mobile robot and / or the task execution end in the fifth image data is determined as the sixth position corresponding to the fifth image data, and a position acquisition request carrying the acquisition time of the fifth image data is sent to the mobile robot to determine the position of the mobile robot and / or the task execution end in the world coordinate system at the acquisition time, as the seventh position corresponding to the fifth image data; based on the sixth and seventh positions corresponding to each fifth image data, a first mapping relationship between the pixel coordinate system corresponding to the camera device and the world coordinate system is determined.

[0192] Optionally, the mobile robot has multiple task processing modes; wherein each task processing mode is used to process task objects of at least one object type.

[0193] The request sending module is further configured to determine the object type of the target task object as the target object type before sending the task execution request carrying the location information to the mobile robot; and send the task execution request carrying the location information and the target object type to the mobile robot so that the mobile robot processes the task object at the location information according to the task processing mode corresponding to the target object type.

[0194] Optionally, there may be multiple mobile robots;

[0195] The request sending module is specifically used to select a mobile robot from a plurality of mobile robots that can process the target task object as the target mobile robot; wherein, the mobile robot that can process the target task object includes: a mobile robot that can process object types including the object type of the task object, and / or a mobile robot whose task area in the target scene includes the location of the target task object, and controls the target mobile robot to process the target task object.

[0196] Optionally, the device further includes:

[0197] The return execution module is used to return to the step of collecting the image data of the target scene as the first image data when no task object of the mobile robot is detected from the first image data, or when the mobile robot has completed the processing of the target task object.

[0198] In the above-described scheme of this application embodiment, when the camera device detects the target task object from the first image data, it indicates that there is a task object of the mobile robot in the target scene. At this time, the camera device sends a task execution request to the mobile robot, which contains the location information of the target task object in the target scene. After receiving the task execution request, the mobile robot can move to the target task object according to the location information and execute the target task, thereby enabling the mobile robot to process the task object in the target scene in a timely manner.

[0199] Corresponding to the task object processing method for mobile robots provided in the above embodiments of this application, such as Figure 8 As shown in the illustration, this application also provides a task object processing device applied to a mobile robot, the device comprising:

[0200] The request receiving module 801 is used to receive a task execution request carrying location information sent by a camera device deployed in a target scene; wherein, the target scene is the scene in which the mobile robot is deployed; the location information is the location information in the target scene determined by the camera device for the target task object in the first image data; the first image data is image data collected by the camera device for the target scene; and the target task object is the task object of the mobile robot detected in the first image data.

[0201] The object processing module 802 is used to move to the target task object according to the location information to execute the target task.

[0202] Optionally, the object processing module is specifically used to perform path planning based on the location information to obtain the path to be moved; to move according to the path to be moved; and to process the task object at the location information after the movement is completed.

[0203] Optionally, the mobile robot has multiple task processing modes; wherein each task processing mode is used to process a task object of at least one object type; the task execution request also carries a target object type, which is the object type of the target task object determined by the camera device;

[0204] The object processing module is specifically used to process the task object at the location information according to the task processing mode corresponding to the target object type after the movement ends.

[0205] Optionally, the object processing module is specifically configured to: if the task execution request is received when the mobile robot is in standby mode, determine a movement path from the current location of the mobile robot to the location information, as the path to be moved; if the task execution request is received when the mobile robot is in a task state, plan a movement path for the mobile robot based on the task state and the location information, as the path to be moved.

[0206] Optionally, the task state includes at least one of a global task state and a sudden task execution state; wherein, the global task state is the state in which the mobile robot is performing global task processing, and the sudden task state is the state in which the mobile robot is performing a task sent by the camera device.

[0207] The object processing module is specifically configured to: if the task execution request is received when the mobile robot is in a global task state, determine whether the location information is located within the executed task area; if it is, immediately determine a movement path from the mobile robot's current location to the location information as the path to be moved, or, after the global task execution ends, determine a movement path from the location at the end of the global task to the location information as the path to be moved; otherwise, use the movement path of the global task as the path to be moved; if the task execution request is received when the mobile robot is in a sudden task state, determine the location from the mobile robot's current location to the location of the task object corresponding to the sudden task state and the movement path of the location information as the path to be moved, when the mobile robot moves to the task object corresponding to the sudden task state; and determine a movement path from the mobile robot's current location to the location information as the path to be moved, while the mobile robot is processing the task object corresponding to the sudden task state.

[0208] Optionally, the object processing module is specifically used to write the location information into a location queue; and based on at least one location contained in the location queue, to plan the movement path of the mobile robot as the path to be moved.

[0209] In the above-described scheme of this application embodiment, when the camera device detects the target task object from the first image data, it indicates that there is a task object of the mobile robot in the target scene. At this time, the camera device sends a task execution request to the mobile robot, which contains the location information of the target task object in the target scene. After receiving the task execution request, the mobile robot can move to the target task object according to the location information and execute the target task, thereby enabling the mobile robot to process the task object in the target scene in a timely manner.

[0210] This application also provides a camera device, such as... Figure 9 As shown, it includes a camera 901, a processor 902, and a machine-readable storage medium 903;

[0211] The camera 901 is used to collect image data of the target scene;

[0212] The machine-readable storage medium 903 stores machine-executable instructions that can be executed by the processor 902, which in turn cause the processor 902 to implement the task object processing method for a camera device provided in the above embodiments of this application.

[0213] It should be emphasized that the processor 902 is prompted by machine-executable instructions to control the camera 901 to acquire image data of the target scene, that is, to acquire image data of the target scene as the first image data.

[0214] This application also provides a mobile robot, such as... Figure 10 As shown, it includes a processor 1001 and a machine-readable storage medium 1002. The machine-readable storage medium 1002 stores machine-executable instructions that can be executed by the processor 1001. The processor 1001 is prompted by the machine-executable instructions to implement the task object processing method for mobile robots provided in the above embodiments of this application.

[0215] Furthermore, the aforementioned camera equipment, camera equipment, or mobile robot may also include a communication bus and / or a communication interface, with the processor, communication interface, and memory communicating with each other through the communication bus.

[0216] The communication bus mentioned in the above-mentioned camera equipment or mobile robot can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0217] The communication interface is used for communication between the aforementioned camera equipment or mobile robot and other devices.

[0218] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0219] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0220] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described task object processing methods.

[0221] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the task object processing methods described in the above embodiments.

[0222] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0223] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0224] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0225] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for processing task objects, characterized in that, The method, applied to camera equipment deployed in a target scene, includes: Image data of the target scene is collected as the first image data; Detect whether the first image data contains a target task object; the target task object is the task object of the mobile robot deployed in the target scene; When the target task object is detected from the first image data, the first position of the target task object in the first image data is determined; Determine whether the first location is located within the target area corresponding to the mobile robot; the target area corresponding to the mobile robot is: the area in the first image data that corresponds to the task area of ​​the mobile robot in the target scene; the method for determining the target area corresponding to the mobile robot includes: acquiring image data collected by the camera device for the target scene as second image data; performing semantic segmentation on the second image data to obtain semantic categories of different image regions in the second image data; determining the image region with a specified semantic category from each image region of the second image data as the target area corresponding to the mobile robot; wherein, the specified semantic category is a preset semantic category of the task execution area of ​​the mobile robot; If the first position is located within the target area, then based on the first mapping relationship between the pixel coordinate system corresponding to the camera device and the world coordinate system where the mobile robot is located, the second position mapped by the first position in the world coordinate system is determined as the position information of the target task object in the target scene. A task execution request carrying the location information is sent to the mobile robot so that the mobile robot moves to the target task object according to the location information to execute the target task.

2. The method according to claim 1, characterized in that, After determining the image region with a specified semantic category from each image region of the second image data as the target region corresponding to the mobile robot, the method further includes: During the inspection of the task execution area by the mobile robot, the camera device is used to detect the mobile robot and / or the task execution end of the mobile robot in real time, so as to determine the image area in the third image data collected by the camera device where the mobile robot and / or the task execution end of the mobile robot is located, which is used as the detection area. Based on the detection area, the target area corresponding to the mobile robot is adjusted.

3. The method according to claim 2, characterized in that, The step of adjusting the target area corresponding to the mobile robot based on the detection area includes: The detection area is incorporated into the target area corresponding to the mobile robot to obtain the adjusted target area; and / or, According to a preset pixel expansion rule, the pixels contained in the detection area are expanded; the expanded detection area is then incorporated into the target area corresponding to the mobile robot to obtain the adjusted target area.

4. The method according to claim 1, characterized in that, The camera parameters of the camera device are adjustable, and the camera parameters include at least one of rotation parameters, movement parameters, and scaling parameters; the pixel coordinate system corresponding to the camera device is the pixel coordinate system of the fourth image data, and the fourth image data is the image data collected by the camera device when it is in the preset camera parameters; The step of determining the second position mapped in the world coordinate system of the first position based on the pre-established first mapping relationship between the pixel coordinate system corresponding to the camera device and the world coordinate system of the mobile robot includes: Feature point matching is performed on the first image data and the fourth image data to obtain the successfully matched feature points in the first image data and the fourth image data, which are used as target feature points; Determine the third position of the target feature point in the first image data and the fourth position in the fourth image data; Based on the third position and the fourth position, a second mapping relationship is determined between the pixel coordinate system of the first image data and the pixel coordinate system of the fourth image data; According to the second mapping relationship, the position of the first position mapped in the pixel coordinate system of the fourth image data is determined as the fifth position; Based on the first mapping relationship between the pixel coordinate system of the fourth image data and the world coordinate system where the mobile robot is located, the position of the fifth position mapped in the world coordinate system is determined, which is then used as the second position mapped by the first position in the world coordinate system.

5. The method according to claim 1, characterized in that, The first mapping relationship between the pixel coordinate system corresponding to the camera device and the world coordinate system where the mobile robot is located is established in the following manner, including: The camera device acquires multiple fifth image data points of the target scene; each fifth image data point includes the mobile robot and / or the task execution terminal of the mobile robot; the task execution terminal is a component of the mobile robot that processes the task object. For each fifth image data, determine the position of the mobile robot and / or the task execution end in the fifth image data as the sixth position corresponding to the fifth image data, and send a position acquisition request carrying the acquisition time of the fifth image data to the mobile robot to determine the position of the mobile robot and / or the task execution end in the world coordinate system at the acquisition time as the seventh position corresponding to the fifth image data; Based on the sixth and seventh positions corresponding to each fifth image data, a first mapping relationship between the pixel coordinate system corresponding to the camera device and the world coordinate system is determined.

6. The method according to claim 1, characterized in that, The mobile robot has multiple task processing modes; each task processing mode is used to process task objects of at least one object type. Before sending the task execution request carrying the location information to the mobile robot, the method further includes: Determine the object type of the target task object, and use it as the target object type; Sending a task execution request carrying the location information to the mobile robot includes: A task execution request carrying the location information and the type of the target object is sent to the mobile robot, so that the mobile robot processes the task object at the location information according to the task processing mode corresponding to the type of the target object.

7. The method according to claim 1, characterized in that, There are multiple mobile robots; Sending a task execution request to the mobile robot for the target task object includes: From a plurality of mobile robots, a mobile robot capable of processing the target task object is selected as the target mobile robot; wherein, the mobile robot capable of processing the target task object includes: a mobile robot whose object type that can be processed includes the object type of the task object, and / or whose task area in the target scene includes the location of the target task object; The target mobile robot is controlled to process the target task object.

8. The method according to claim 1, characterized in that, The method further includes: When no task object of the mobile robot is detected from the first image data, or when the mobile robot has completed processing for the target task object, the process returns to the step of collecting image data of the target scene as the first image data.

9. A method for processing task objects, characterized in that, Applied to mobile robots, the method includes: The system receives a task execution request carrying location information from a camera device deployed in a target scene. The target scene is the scene in which the mobile robot is deployed. The location information is the location information within the target scene determined by the camera device for a target task object in first image data. The first image data is image data collected by the camera device for the target scene. The target task object is the task object of the mobile robot detected in the first image data. The location information is determined as follows: when the target task object is detected from the first image data, a first position of the target task object in the first image data is determined; it is then determined whether the first position is located within the target area corresponding to the mobile robot. The target area corresponding to the mobile robot is defined as the area in the first image data corresponding to the task area of ​​the mobile robot in the target scene. The method for determining the target area corresponding to the mobile robot includes: acquiring image data collected by the camera device for the target scene as second image data; performing semantic segmentation on the second image data to obtain semantic categories of different image regions in the second image data; determining image regions with a specified semantic category from each image region of the second image data as the target area corresponding to the mobile robot; wherein, the specified semantic category is a preset semantic category of the task execution area of ​​the mobile robot; if the first position is located within the target area, then based on a pre-established first mapping relationship between the pixel coordinate system corresponding to the camera device and the world coordinate system where the mobile robot is located, determining the second position mapped by the first position in the world coordinate system as the position information of the target task object in the target scene; Move to the target task object according to the location information to execute the target task.

10. The method according to claim 9, characterized in that, The step of moving to the target task object according to the location information to execute the target task includes: Based on the location information, a path is planned to obtain the path to be moved. Move according to the path to be moved, and process the task object at the location information after the move is completed.

11. The method according to claim 10, characterized in that, The mobile robot has multiple task processing modes; each task processing mode is used to process task objects of at least one object type; the task execution request also carries the target object type, which is the object type of the target task object determined by the camera device; The processing of the task object at the location information after the movement ends includes: After the movement is completed, the task object at the location information is processed according to the task processing mode corresponding to the target object type.

12. The method according to claim 10, characterized in that, The step of route planning based on the location information to obtain the path to be moved includes: If the task execution request was received when the mobile robot was in standby mode, then a movement path from the current location of the mobile robot to the location information is determined as the path to be moved. If the task execution request is received by the mobile robot while it is in a task state, then based on the task state and the location information, the mobile robot's movement path is planned as the path to be moved.

13. The method according to claim 12, characterized in that, The task state includes at least one of a global task state and a sudden task state; wherein, the global task state is the state in which the mobile robot is performing global task processing, and the sudden task state is the state in which the mobile robot is performing a task sent by the camera device. If the task execution request is received by the mobile robot while it is in a task state, then based on the task state and the location information, a movement path for the mobile robot is planned as the path to be moved, including: If the task execution request is received when the mobile robot is in a global task state, then it is determined whether the location information is within the area of ​​the executed task; if it is, then a movement path from the current location of the mobile robot to the location information is immediately determined as the path to be moved, or, after the global task execution ends, a movement path from the location at the end of the global task to the location information is determined as the path to be moved; otherwise, the movement path of the global task is taken as the path to be moved. If the task execution request is received when the mobile robot is in a sudden task state, then when the mobile robot moves to the task object corresponding to the sudden task state, a movement path from the current position of the mobile robot to the position of the task object corresponding to the sudden task state and the location information are determined as the movement path to be moved; during the process of the mobile robot processing the task object corresponding to the sudden task state, a movement path from the current position of the mobile robot to the location information is determined as the movement path to be moved.

14. The method according to claim 10, characterized in that, The step of route planning based on the location information to obtain the path to be moved includes: Write the location information into the location queue; Based on at least one position contained in the position queue, the mobile robot's movement path is planned as the path to be moved.

15. A task object processing system, characterized in that, The task object processing system includes: camera equipment and a mobile robot; The camera device is deployed within the target scene and is used to collect image data of the target scene as first image data; detect whether the first image data contains a target task object; the target task object is the task object of the mobile robot deployed within the target scene; when the target task object is detected from the first image data, determine the first position of the target task object in the first image data; determine whether the first position is located within the target area corresponding to the mobile robot; the target area corresponding to the mobile robot is: the area in the first image data corresponding to the task area of ​​the mobile robot in the target scene; the method of determining the target area corresponding to the mobile robot includes: acquiring image data collected by the camera device for the target scene as second image data. Image data; semantic segmentation is performed on the second image data to obtain the semantic categories of different image regions in the second image data; from each image region of the second image data, the image region with the specified semantic category is determined as the target region corresponding to the mobile robot; wherein, the specified semantic category is the semantic category of the preset task execution region of the mobile robot; if the first position is located within the target region, then based on the first mapping relationship between the pixel coordinate system corresponding to the camera device and the world coordinate system where the mobile robot is located, the second position mapped in the world coordinate system of the first position is determined as the position information of the target task object in the target scene; a task execution request carrying the position information is sent to the mobile robot; The mobile robot is used to receive the task execution request sent by the camera device, and move to the target task object according to the location information to execute the target task.

16. A camera device, characterized in that, Includes cameras, processors, and machine-readable storage media; The camera is used to acquire image data of the target scene; The machine-readable storage medium stores machine-executable instructions that can be executed by the processor, which is prompted by the machine-executable instructions to implement the method of any one of claims 1-8.

17. A mobile robot, characterized in that, The method includes a processor and a machine-readable storage medium; the machine-readable storage medium stores machine-executable instructions that can be executed by the processor, which is prompted by the machine-executable instructions to perform the method of any one of claims 9-14.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-8 or 9-14.

Images