Image processing method and device based on visual robot, equipment and medium

Abstract

The invention relates to an image processing method and device based on a visual robot, equipment and a medium. The method comprises the steps that a sampling image output by a camera of the visual robot is acquired; determining a plurality of first image areas including a set object in the sampling image and image feature information of the set object in the plurality of first image areas; screening target image feature information meeting a set condition from the multiple pieces of image feature information; and according to target object feature information in the target image feature information, setting operation parameters of the visual robot mechanical arm.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to an image processing method, apparatus, device, and medium based on a vision robot. Background Technology

[0002] With the rapid development of visual robots and image processing technology, visual robots can be used as harvesting robots in the agricultural field to identify and process harvesting targets. Currently, in order to improve the working efficiency of visual robots, existing visual robots immediately process the harvesting target after identification. However, existing visual robots have difficulty adjusting their operating parameters according to the working environment and the maturity of the target, which affects the harvesting accuracy. Summary of the Invention

[0003] One objective of this disclosure is to provide a new technical solution for image processing based on vision robots.

[0004] According to a first aspect of this disclosure, an image processing method based on a vision robot is provided, the method comprising: Acquire sampled images output by the camera of the vision robot; Using a pre-set deep learning network model, multiple first image regions containing a specified object in the sampled image are determined, along with image feature information of the specified object within these multiple first image regions. Filter target image feature information that meets the set conditions from multiple image feature information; The operating parameters of the vision robot arm are set based on the target object feature information in the target image feature information.

[0005] Optionally, before acquiring the sampled image output by the camera of the visual robot, the method further includes: Obtain training samples; By using a pre-set image annotation tool, multi-task semantic annotation is performed on the training samples to obtain the multi-task parsing dataset corresponding to the sampled image; Based on the multi-task parsing dataset, the deep learning network model is trained to obtain a trained deep learning network model.

[0006] Optionally, the set conditions include image attribute exclusion conditions and object attribute exclusion conditions; the filtering of target image feature information that meets the set conditions from multiple image feature information includes: Among the multiple image feature information, image feature information that meets the image attribute exclusion conditions is selected as the first image feature information; In the first image feature information, the first image feature information that meets the object attribute exclusion conditions is selected as the target image feature information.

[0007] Optionally, the step of filtering image feature information that meets the image attribute exclusion conditions from the plurality of image feature information to use as the first image feature information includes: Extract second image feature information from among the multiple image feature information that has obstruction annotation information representing the designated object; Among the multiple image feature information, the second image feature information is filtered out to obtain image feature information that meets the image attribute exclusion conditions, which is then used as the first image feature information.

[0008] Optionally, the step of filtering the first image feature information that meets the object attribute exclusion conditions as target image feature information includes: For each of the first image feature information, determine the two-dimensional position and depth value of the set object at the center point of the corresponding first image region, as well as the geometric features of the set object in the corresponding first image region; input the two-dimensional position, the depth value, and the geometric features into a preset distance determination model to obtain the three-dimensional dimensions of the set object; Among the three-dimensional dimensions of the set object reflected by each of the first image feature information, a target three-dimensional feature that matches the object attribute exclusion condition and a first image feature information corresponding to the target three-dimensional feature are determined as target image feature information.

[0009] Optionally, the distance determination model includes: in, This indicates the three-dimensional position of the center point of the set object. This indicates the three-dimensional position of the edge points of the set object. pixel_to_point This is a transformation function that converts image pixel coordinates to camera 3D space coordinates. instrincs The internal parameters of the camera are set. This indicates the two-dimensional position of the center point of the set object. This indicates the two-dimensional position of the edge point of the set object. depth This represents the depth value.

[0010] Optionally, before setting the operating parameters of the visual robot arm based on the target object feature information in the target image feature information, the method further includes: The depth value reflecting the center point of the set object and the three-dimensional size of the set object in the target image feature information are input into a preset priority determination model to obtain the priority of the target image feature information. The priority, the two-dimensional position of the center point of the set object and the three-dimensional size of the set object in the target image feature information are used as the target object feature information in the target image feature information. The step of setting the operating parameters of the visual robot arm based on the target object feature information in the target image feature information includes: The operating sequence of the vision robot arm is set according to the priority reflected by the feature information of each target object; Based on the two-dimensional position of the center point of the target object reflected by the feature information of each target object and the three-dimensional size of the target object, the operating pose of the vision robot arm at each operating node is set; wherein, each operating node is a node in the operating sequence.

[0011] According to a second aspect of this disclosure, an image processing apparatus based on a vision robot is also provided, the apparatus comprising: The acquisition module is used to acquire sampled images output by the camera of the vision robot; The determining module is used to determine multiple first image regions containing a specified object in the sampled image and image feature information of the specified object in the multiple first image regions; The filtering module is used to filter target image feature information that meets set conditions from multiple image feature information; The setting module is used to set the operating parameters of the vision robot arm based on the target object feature information in the target image feature information.

[0012] According to a third aspect of this disclosure, an electronic device is also provided, including a memory and a processor, the memory being used to store a computer program; the processor being used to execute the computer program to implement the method according to a first aspect of this disclosure.

[0013] According to a fourth aspect of this disclosure, a computer-readable storage medium is also provided, on which a computer program is stored, the computer program implementing the method according to a first aspect of this disclosure when executed by a processor.

[0014] According to a fifth aspect of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the method described according to a first aspect of this disclosure.

[0015] One beneficial effect of this disclosure is that the image processing method based on a vision robot provided by the present invention can acquire sampled images through the camera of the vision robot, determine a first image region containing a set object and image feature information of the set object in the first image region, and then filter out target image feature information that meets set conditions. Based on the target object feature information in the target image feature information, the operating parameters of the vision robot arm can be set. By filtering out target image feature information that meets set conditions, the vision robot arm can be controlled to process the set object appearing in the target image feature information, which can reduce the occurrence of processing set objects with obstacles or that are not yet mature, effectively improving the picking accuracy of the vision robot.

[0016] Other features and advantages of the embodiments of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the present disclosure and, together with their description, serve to explain the principles of the embodiments of the present disclosure.

[0018] Figure 1 It is a schematic diagram of the composition structure of a vision robot that can be applied according to one embodiment; Figure 2 This is a flowchart illustrating an image processing method based on a vision robot according to one embodiment; Figure 3 This is a block diagram of an image processing apparatus based on a vision robot according to one embodiment; Figure 4 This is a schematic diagram of the hardware structure of an electronic device according to one embodiment. Detailed Implementation

[0019] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0020] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0021] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0022] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0023] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0024] <System Implementation> Figure 1 This is a schematic diagram of the structural composition of a vision robot applicable to one embodiment. For example... Figure 1 As shown, the visual robot may include a camera 10, a controller 20, and a robotic arm 30, and is used in agricultural scenarios.

[0025] The camera 10 can be mounted on the base of the vision robot or on the end effector of the vision robot.

[0026] The robotic arm 30 can be mounted on the base of the vision robot. The robotic arm 30 can be a multi-joint structure, capable of multi-degree-of-freedom movement, in order to grasp or transport objects.

[0027] The controller 20 can be electrically connected to the camera 10 and the robotic arm 30 respectively. The controller 20 can obtain sampled images through the camera 10 and can also adjust the pose of the robotic arm 30 to complete different work tasks.

[0028] In the embodiments of this disclosure, the controller 20 memory stores a computer program that controls the controller 20 processor to operate according to the image processing method based on a vision robot according to any embodiment. Those skilled in the art can design the computer program based on the scheme of the embodiments of this disclosure. How the computer program controls the processor to operate is well known in the art and will not be described in detail here.

[0029] <Method Implementation> Figure 2 This is a flowchart illustrating an image processing method based on a vision robot according to one embodiment. The implementing entity is, for example, a... Figure 1 Controller 20.

[0030] like Figure 2 As shown, the image processing method based on a vision robot in this embodiment may include the following steps S210 to S240: Step S210: Acquire the sampled image output by the camera of the vision robot.

[0031] In this embodiment, the sampled image can be an RGB image captured by the camera of a vision robot in an agricultural greenhouse. The sampled image may contain a set of objects that are occluded by different obstacles and / or unoccluded set of objects. The set of objects here is, for example, fruit, and the set of objects will be described as fruit in the following description.

[0032] Step S220: Determine multiple first image regions containing the specified object in the sampled image, as well as image feature information of the specified object in the multiple first image regions.

[0033] In some embodiments, step S220 may include the following steps S310 and S330: Step S310: Obtain training samples.

[0034] Step S320: Using a pre-set image annotation tool, perform multi-task semantic annotation on the training samples to obtain the multi-task parsing dataset corresponding to the sampled images.

[0035] In this embodiment, multi-task semantic annotation of images is performed using image annotation tools, including global semantic segmentation annotation, object instance segmentation annotation, and object key point annotation, in order to construct a multi-task parsing dataset for images.

[0036] In this embodiment, the object is set as fruit, and the image annotation tool is, for example, Labelme. This tool is used to perform multi-task semantic annotation on the image. Global semantic segmentation annotation includes categories such as fruit (ripe, immature), main obstacles (trunk, lateral branches, hanging branches), and background. Object instance segmentation annotation can divide the sampled image containing each fruit into multiple image blocks. Object keypoints can include the end of the fruit stem, the center point of the fruit, and the tail end of the fruit; object keypoint annotation can label these object keypoints. Through global semantic segmentation annotation, object instance segmentation annotation, and object keypoint annotation, a multi-task parsing dataset is constructed.

[0037] In this embodiment, the controller is further configured to reconstruct the YOLOV8 (You Only Look Once) network. By sharing feature extraction layers, customizing task-specific layers, and jointly optimizing the multi-task loss function, the three tasks of global image semantic segmentation, object instance segmentation, and object keypoint detection are integrated into a unified deep learning network model. Specifically, the deep learning network model can be, for example, the Darknet-53 network model, used as a shared feature extraction network for the image; the detection head is modified to be a multi-task detection head capable of simultaneously performing global image semantic segmentation, object instance segmentation, and object keypoint detection; and the weighted sum of the global semantic segmentation loss, object instance segmentation loss, and object keypoint detection loss is used as the network's loss value. Thus, while achieving lightweight integration of multiple tasks, the reconstructed YOLOV8 (You Only Look Once) network enables multi-task parsing of the sampled image.

[0038] Step S320: Based on the multi-task parsing dataset, train the deep learning network model to obtain the trained deep learning network model.

[0039] In this embodiment, using the trained deep learning network model, a multi-task parsing dataset of training samples is first obtained. Then, the pixel region corresponding to each fruit segmentation mask on the semantic segmentation result map is searched and analyzed, that is, multiple first image regions containing the set object are determined in the sampled image. The image feature information in the first image region may include whether the fruit is ripe, whether it is occluded by an obstacle, which image block it corresponds to, the end of the fruit stem, the center point of the fruit, and the position of the tail end of the fruit in the first image region, etc.

[0040] Step S230: Filter target image feature information that meets the set conditions from multiple image feature information.

[0041] In some embodiments, in order to enable a visual robot to pick easily harvested and relatively ripe fruits through sampled images, the target image feature information can be filtered based on the image background of the sampled image and the object attributes of a specified object in the sampled image. The set conditions include image attribute exclusion conditions and object attribute exclusion conditions; this step S230 may include the following steps S410 and S420: Step S410: Among multiple image feature information, image feature information that meets the image attribute exclusion conditions is selected as the first image feature information.

[0042] Step S420: In the first image feature information, filter the first image feature information that meets the object attribute exclusion conditions to use as the target image feature information.

[0043] In this embodiment, by setting image attribute exclusion conditions and object attribute exclusion conditions, fruits that are not difficult to pick and are relatively mature, as well as the target image feature information corresponding to the fruits, can be filtered out.

[0044] In some embodiments, step S410 may include the following steps S510 and S520: Step S510: Extract the second image feature information from multiple image feature information that has obstruction label information representing the set object.

[0045] Step S520: Among multiple image feature information, the second image feature information is filtered out to obtain image feature information that meets the image attribute exclusion conditions, which is then used as the first image feature information.

[0046] In this embodiment, for the image feature information of each first image region, it can be determined whether the fruit segmentation mask has annotations indicating obstacles such as hanging lines or branches on the semantic segmentation map; that is, whether the image feature information contains obstruction annotation information representing a set object. Furthermore, if a first image region corresponds to an annotation indicating obstruction, the first image region is filtered out. If a first image region does not correspond to an annotation indicating obstruction, the first image region and its corresponding image feature information are retained, thus obtaining image feature information that meets the image attribute exclusion conditions, which is used as the first image feature information.

[0047] In this embodiment, by selecting the image feature information corresponding to fruits without picking obstacles, it is possible to filter out the set objects that are blocked by multiple objects and select the set objects that are not blocked in an unstructured picking environment, thereby improving the visual obstacle avoidance capability of the visual robot and ensuring the safety of the picking operation.

[0048] In some embodiments, step S420 may include the following steps S610 and S620: Step S610: For each first image feature information, determine the two-dimensional position and depth value of the set object at the center point of the corresponding first image region, as well as the geometric features of the set object in the corresponding first image region; input the two-dimensional position, depth value, and geometric features into a preset distance determination model to obtain the three-dimensional dimensions of the set object.

[0049] Step S620: Among the three-dimensional dimensions of the set object reflected by each first image feature information, determine the target three-dimensional feature that matches the object attribute exclusion condition and the first image feature information corresponding to the target three-dimensional feature, and use them as target image feature information.

[0050] In this embodiment, for each first image feature, the minimum circumscribed circle radius of the target object can be determined as the geometric feature of the target object in the corresponding first image region. Combined with the two-dimensional position of the center point of the minimum circumscribed circle (i.e., the two-dimensional position of the center point of the target object in the corresponding first image region) and the depth value of the center point of the target object in the corresponding first image region, a three-dimensional coordinate transformation is performed to estimate the three-dimensional dimensions of the target object in three-dimensional space, i.e., the actual size and actual radius of the target object. Furthermore, the object attribute exclusion condition can be to select the target three-dimensional feature from multiple three-dimensional dimensions that has the largest area and the smallest distance from the current position of the vision machine.

[0051] In this embodiment, by providing the visual robot with information such as the size and pose estimation of the set object, the accuracy and intelligence of the visual robot during operation can be further improved, enabling the visual robot to harvest without damage.

[0052] In this embodiment, the distance determination model includes: in, This indicates the three-dimensional position of the center point of the set object. This indicates the 3D position of the edge points of the set object. `pixel_to_point` is the conversion function for converting image pixel coordinates to camera 3D space coordinates, and `instrincs` is the set intrinsic parameter of the camera. This indicates the two-dimensional position of the center point of the set object. This indicates the two-dimensional position of a point on the edge of an object, where depth represents the depth value. R This represents the distance between the center point of the defined object's edge and the defined edge point itself, i.e., the actual radius. Here, the defined object's edge point refers to a virtual edge point.

[0053] Step S240: Set the operating parameters of the vision robot arm based on the target object feature information in the target image feature information.

[0054] In some embodiments, prior to step S240, the method further includes the following step S710: Step S710: Input the depth value reflecting the center point of the set object and the three-dimensional size of the set object in the target image feature information into the preset priority determination model to obtain the priority of the target image feature information, and use the priority, the two-dimensional position of the center point of the set object and the three-dimensional size of the set object in the target image feature information as the target object feature information in the target image feature information.

[0055] In this embodiment, the priority determination model includes: Where R represents the actual radius, D This represents the depth value.

[0056] In this embodiment, the two-dimensional position of the center point of the target object in the target image feature information can be coordinates (u0, v0), and the three-dimensional size of the target object can be the actual radius mentioned above.

[0057] Based on this, step S240 may include the following steps S720 and S730: Step S720: Set the running sequence of the vision robot arm according to the priority reflected by the feature information of each target object.

[0058] In this embodiment, after obtaining the priority reflected by the feature information of each target object, the highest priority value can be determined, and the corresponding running order can be obtained according to the order from largest to smallest. For example, if the priorities are 30, 40, 60, and 50, then the running order is 60-50-40-30.

[0059] Step S730: Based on the two-dimensional position of the center point of the target object and the three-dimensional size of the target object reflected by the feature information of each target object, set the running pose of the vision robot arm at each running node; wherein, each running node is a node in the running sequence.

[0060] In this embodiment, once the two-dimensional position of the center point of the target object and the three-dimensional dimensions of the target object are determined, the vision robot arm can adjust its corresponding operating pose to process the target object. The vision robot arm's pose adjustment based on the target's image position and three-dimensional dimensions is existing technology and will not be described in detail here.

[0061] In this embodiment, by setting a priority, the optimal target in the sampled image is processed first, which effectively improves the picking accuracy of the visual robot.

[0062] <Equipment Example 1> Figure 3 This is a schematic block diagram of an image processing apparatus based on a vision robot according to one embodiment. Figure 3 As shown, the image processing device 300 based on vision robots may include: The acquisition module 310 is used to acquire sampled images output by the camera of the vision robot; The determining module 320 is used to determine multiple first image regions containing a set object in the sampled image and image feature information of the set object in the multiple first image regions; The filtering module 330 is used to filter target image feature information that meets set conditions from multiple image feature information; The setting module 340 is used to set the operating parameters of the vision robot arm based on the target object feature information in the target image feature information.

[0063] Optionally, the image processing device 300 based on the vision robot also includes a training module for acquiring training samples; performing multi-task semantic annotation on the training samples using a pre-set image annotation tool to obtain a multi-task parsing dataset corresponding to the sampled image; and training the deep learning network model based on the multi-task parsing dataset to obtain a trained deep learning network model.

[0064] Optionally, the filtering module 330 is further configured to filter image feature information that meets the image attribute exclusion conditions from multiple image feature information to serve as first image feature information; and to filter first image feature information that meets the object attribute exclusion conditions from the first image feature information to serve as target image feature information.

[0065] Optionally, the filtering module 330 is further configured to extract second image feature information that has obstruction label information representing a set object from multiple image feature information; and to filter out the second image feature information from the multiple image feature information to obtain image feature information that meets the image attribute exclusion conditions, which is then used as the first image feature information.

[0066] Optionally, the filtering module 330 is further configured to, for each first image feature information, determine the two-dimensional position and depth value of the set object at the center point of the corresponding first image region, as well as the geometric features of the set object in the corresponding first image region; input the two-dimensional position, depth value, and geometric features into a preset distance determination model to obtain the three-dimensional dimensions of the set object; and, among the three-dimensional dimensions of the set object reflected by each first image feature information, determine the target three-dimensional features that match the object attribute exclusion conditions and the first image feature information corresponding to the target three-dimensional features, as the target image feature information.

[0067] Optionally, the image processing device 300 based on the vision robot further includes a priority determination module, which is used to input the depth value reflecting the center point of the set object and the three-dimensional size of the set object in the target image feature information into a preset priority determination model to obtain the priority of the target image feature information, and use the priority, the two-dimensional position of the center point of the set object and the three-dimensional size of the set object in the target image feature information as the target object feature information in the target image feature information.

[0068] The setting module 340 is also used to set the running sequence of the vision robot arm according to the priority reflected by the feature information of each target object; and to set the running pose of the vision robot arm at each running node according to the two-dimensional position of the center point of the set object and the three-dimensional size of the set object reflected by the feature information of each target object; wherein, each running node is a node in the running sequence.

[0069] The image processing device 300 based on the vision robot can be Figure 1 The controller 20 in the middle.

[0070] <Equipment Example 2> Figure 4 This is a schematic diagram of the hardware structure of an electronic device according to another embodiment.

[0071] like Figure 4 As shown, the electronic device 400 includes a processor 410 and a memory 420, the memory 420 for storing an executable computer program, and the processor 410 for executing methods as described in any of the above method embodiments under the control of the computer program.

[0072] The electronic device can be Figure 1 The controller 20 in the middle.

[0073] Each module of the image processing device 300 based on the vision robot described above can be implemented by the processor 410 in this embodiment executing the computer program stored in the memory 420, or it can be implemented by other structures, which are not limited here.

[0074] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0075] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0076] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0077] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of the present invention.

[0078] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0079] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0080] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0081] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0082] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. An image processing method based on a vision robot, characterized in that, The method includes: Acquire sampled images output by the camera of the vision robot; Using a pre-set deep learning network model, multiple first image regions containing a specified object in the sampled image are determined, along with image feature information of the specified object within these multiple first image regions. Filter target image feature information that meets the set conditions from multiple image feature information; The operating parameters of the vision robot arm are set based on the target object feature information in the target image feature information.

2. The method according to claim 1, characterized in that, Before acquiring the sampled image output by the camera of the vision robot, the method further includes: Obtain training samples; By using a pre-set image annotation tool, multi-task semantic annotation is performed on the training samples to obtain the multi-task parsing dataset corresponding to the sampled image; Based on the multi-task parsing dataset, the deep learning network model is trained to obtain a trained deep learning network model.

3. The method according to claim 1, characterized in that, The set conditions include image attribute exclusion conditions and object attribute exclusion conditions; the filtering of target image feature information that meets the set conditions from multiple image feature information includes: Among the multiple image feature information, image feature information that meets the image attribute exclusion conditions is selected as the first image feature information; In the first image feature information, the first image feature information that meets the object attribute exclusion conditions is selected as the target image feature information.

4. The method according to claim 3, characterized in that, The step of filtering image feature information that meets the image attribute exclusion conditions from multiple image feature information to use as the first image feature information includes: Extract second image feature information from among the multiple image feature information that has obstruction annotation information representing the designated object; Among the multiple image feature information, the second image feature information is filtered out to obtain image feature information that meets the image attribute exclusion conditions, which is then used as the first image feature information.

5. The method according to claim 3, characterized in that, The step of filtering first image feature information that meets the object attribute exclusion conditions from the first image feature information to use as target image feature information includes: For each of the first image feature information, determine the two-dimensional position and depth value of the set object at the center point of the corresponding first image region, as well as the geometric features of the set object in the corresponding first image region; input the two-dimensional position, the depth value, and the geometric features into a preset distance determination model to obtain the three-dimensional dimensions of the set object; Among the three-dimensional dimensions of the set object reflected by each of the first image feature information, a target three-dimensional feature that matches the object attribute exclusion condition and a first image feature information corresponding to the target three-dimensional feature are determined as target image feature information.

6. The method according to claim 5, characterized in that, The distance determination model includes: in, This indicates the three-dimensional position of the center point of the set object. This indicates the three-dimensional position of the edge points of the set object. pixel_to_point This is a transformation function that converts image pixel coordinates to camera 3D space coordinates. instrincs The internal parameters of the camera are set. This indicates the two-dimensional position of the center point of the set object. This indicates the two-dimensional position of the edge point of the set object. depth This represents the depth value.

7. The method according to claim 5, characterized in that, Before setting the operating parameters of the vision robot arm based on the target object feature information in the target image feature information, the method further includes: The depth value reflecting the center point of the set object and the three-dimensional size of the set object in the target image feature information are input into a preset priority determination model to obtain the priority of the target image feature information. The priority, the two-dimensional position of the center point of the set object and the three-dimensional size of the set object in the target image feature information are used as the target object feature information in the target image feature information. The step of setting the operating parameters of the visual robot arm based on the target object feature information in the target image feature information includes: The operating sequence of the vision robot arm is set according to the priority reflected by the feature information of each target object; Based on the two-dimensional position of the center point of the target object reflected by the feature information of each target object and the three-dimensional size of the target object, the operating pose of the vision robot arm at each operating node is set; wherein, each operating node is a node in the operating sequence.

8. An image processing device based on a vision robot, characterized in that, The device includes: The acquisition module is used to acquire sampled images output by the camera of the vision robot; The determining module is used to determine multiple first image regions containing a specified object in the sampled image and image feature information of the specified object in the multiple first image regions; The filtering module is used to filter target image feature information that meets set conditions from multiple image feature information; The setting module is used to set the operating parameters of the vision robot arm based on the target object feature information in the target image feature information.

9. An electronic device, characterized in that, The system includes a memory and a processor, the memory being used to store a computer program; the processor being used to execute the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images