Object recognition positioning method, system, computer and readable storage medium

Abstract

The application provides an object recognition positioning method, system, computer and readable storage medium, the method comprises the following steps: obtaining order information, judging whether the order name in the order information is consistent with the production name; if yes, unload the original neural network model weight, and load the target neural network model weight, input the target neural network model weight into the neural network model; obtaining the entity image corresponding to the order information, and performing segmentation processing on the entity image through the neural network model to obtain the classification result, regression frame result and mask result of the entity image; according to the classification result, it is judged whether the target object exists in the entity image, if yes, the space pose of the target object is calculated through the regression frame result and the mask result. Through the above-mentioned mode, the composite robot and the flexible automatic production line can quickly identify and grasp different types of objects, and the working efficiency and the intelligent degree of the machine equipment are greatly improved.

Description

Technical Field

[0001] This invention relates to the field of composite robot technology, and in particular to an object recognition and localization method, system, computer, and readable storage medium. Background Technology

[0002] A robot is an intelligent machine capable of semi-autonomous or fully autonomous operation. It possesses basic characteristics such as perception, decision-making, and execution, and can assist or even replace humans in performing dangerous, arduous, and complex tasks, improving work efficiency and quality, serving human life, and expanding or extending the scope of human activities and capabilities.

[0003] Among them, the composite robot is a new type of robot that integrates the functions of both hands and feet. It can replace traditional robots to complete many flexible tasks, greatly improving the automation and intelligence of the working environment.

[0004] However, existing composite robots and robotic arms in flexible automated production lines require vision guidance systems to grasp objects. Furthermore, these vision guidance systems need to be customized by staff based on the different application scenarios of the current machines, which limits the use of the machines and hinders their large-scale promotion and use. Summary of the Invention

[0005] Based on this, the purpose of this invention is to provide an object recognition and positioning method, system, computer, and readable storage medium to solve the problem that the existing composite robots have significant limitations in use, which hinders the large-scale promotion and use of composite robots.

[0006] The first aspect of this invention provides an object recognition and localization method, the method comprising:

[0007] When order information is obtained, it is determined whether the order name in the order information is consistent with the current production name;

[0008] If it is determined that the order name in the order information is inconsistent with the current production name, then the original neural network model weights corresponding to the production name are unloaded, the target neural network model weights corresponding to the order name are loaded from the model weight database, the supporting information corresponding to the order name is read out, and the target neural network model weights are input into the preset neural network model.

[0009] Obtain the entity image corresponding to the order information, and perform segmentation processing on the entity image through the preset neural network model to obtain the classification result, regression box result and mask result corresponding to the entity image;

[0010] Based on the classification results, determine whether a target object exists in the entity image;

[0011] If the target object is determined to exist in the entity image, the spatial pose of the target object is calculated using the regression box result and the mask result. The spatial pose includes the three-dimensional coordinates and three-dimensional angles of the target object.

[0012] The beneficial effects of this invention are as follows: By acquiring order information in real time and determining whether the order name in the current order information matches the production name of the current object being produced; if not, the original neural network model weights corresponding to the current production name are unloaded, and the target neural network model weights corresponding to the current order name are loaded from the model weight database, the supporting information corresponding to the current order name is read, and the target neural network model weights are input into the preset neural network model; furthermore, the current entity image is acquired, and the current entity image is segmented using the aforementioned neural network model to obtain the corresponding classification results, regression box results, and mask results; finally, it is determined whether a target object exists in the current entity image based on the classification results; if so, the spatial pose of the current target object is calculated using the regression box results and mask results. Through the above method, the neural network model weights can be switched in real time according to the order name in the order information, and the acquired entity image can be segmented based on the preset neural network model. Finally, the spatial pose of the target object can be easily and quickly obtained based on the processing results, thereby enabling the composite robot to quickly grasp different types of objects, significantly improving the working efficiency of the composite robot, and making it suitable for widespread promotion and use.

[0013] Preferably, the method further includes:

[0014] If it is determined that the target object does not exist in the entity image, a second entity image corresponding to the order information is obtained, and the second entity image is segmented using the preset neural network model until the target object appears.

[0015] Preferably, the step of calculating the spatial pose of the target object using the regression box result and the mask result includes:

[0016] The target point cloud is calculated based on the mask result and the depth image corresponding to the target object, and the corresponding target pose is matched in the preset point cloud template based on the target point cloud.

[0017] The three-dimensional angles corresponding to the target object are calculated based on the target posture.

[0018] Preferably, the step of calculating the spatial pose of the target object using the regression box result and the mask result further includes:

[0019] The three-dimensional coordinates of the target object are calculated based on the preset algorithm and the regression box results, and the three-dimensional angle and the three-dimensional coordinates are fused to generate the spatial pose of the target object.

[0020] Preferably, after the step of calculating the spatial pose of the target object using the regression box result and the mask result, the method further includes:

[0021] The spatial pose is transmitted to the controller of the robotic arm, so that the controller controls the robotic arm to grasp the target object according to the spatial pose.

[0022] A second aspect of this invention provides an object recognition and positioning system, the system comprising:

[0023] The first judgment module is used to determine whether the order name in the order information is consistent with the current production name when the order information is obtained;

[0024] The first execution module is used to, if it is determined that the order name in the order information is inconsistent with the current production name, unload the original neural network model weights corresponding to the production name, load the target neural network model weights corresponding to the order name from the model weight database, read the supporting information corresponding to the order name, and input the target neural network model weights into the preset neural network model;

[0025] The processing module is used to acquire the entity image corresponding to the order information, and to perform segmentation processing on the entity image through the preset neural network model to obtain the classification result, regression box result and mask result corresponding to the entity image;

[0026] The second judgment module is used to determine whether a target object exists in the entity image based on the classification result;

[0027] The second execution module is used to calculate the spatial pose of the target object by means of the regression box result and the mask result if the target object is determined to exist in the entity image. The spatial pose includes the three-dimensional coordinates and three-dimensional angles of the target object.

[0028] In the aforementioned object recognition and positioning system, the object recognition and positioning system further includes a loop module, which is specifically used for:

[0029] If it is determined that the target object does not exist in the entity image, a second entity image corresponding to the order information is obtained, and the second entity image is segmented using the preset neural network model until the target object appears.

[0030] In the aforementioned object recognition and positioning system, the second execution module is specifically used for:

[0031] The target point cloud is calculated based on the mask result and the depth image corresponding to the target object, and the corresponding target pose is matched in the preset point cloud template based on the target point cloud.

[0032] The three-dimensional angles corresponding to the target object are calculated based on the target posture.

[0033] In the aforementioned object recognition and positioning system, the second execution module is further specifically used for:

[0034] The three-dimensional coordinates of the target object are calculated based on the preset algorithm and the regression box results, and the three-dimensional angle and the three-dimensional coordinates are fused to generate the spatial pose of the target object.

[0035] In the aforementioned object recognition and positioning system, the object recognition and positioning system further includes a transmission module, which is specifically used for:

[0036] The spatial pose is transmitted to the controller of the robotic arm, so that the controller controls the robotic arm to grasp the target object according to the spatial pose.

[0037] A third aspect of the present invention provides a computer, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the object recognition and localization method as described above.

[0038] A fourth aspect of the present invention provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the object recognition and localization method as described above.

[0039] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0040] Figure 1 A flowchart of the object recognition and localization method provided in the first embodiment of the present invention;

[0041] Figure 2 This is a structural block diagram of the object recognition and positioning system provided in the second embodiment of the present invention.

[0042] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0043] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0044] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0046] Please see Figure 1 The image shows the object recognition and localization method provided in the first embodiment of the present invention. The object recognition and localization method provided in this embodiment can switch the weights of the neural network model in real time according to the order name in the order information, and perform segmentation processing on the acquired entity image based on the preset neural network model. Finally, the spatial pose of the target object can be obtained simply and quickly based on the processing result, thereby enabling the composite robot to quickly grasp different types of objects, greatly improving the working efficiency of the composite robot, and is suitable for wide-ranging promotion and use.

[0047] Specifically, the object recognition and localization method provided in this embodiment includes the following steps:

[0048] Step S10: When order information is obtained, determine whether the order name in the order information is consistent with the current production name;

[0049] Specifically, in this embodiment, it should first be noted that the object recognition and positioning method provided in this embodiment is specifically applied in situations where composite robots or flexible automated production lines need to frequently switch target objects during actual operation, and at the same time need to grasp objects through visual guidance.

[0050] In addition, in this embodiment, the aforementioned composite robot or flexible automated production line needs to perform corresponding tasks according to actual production orders during the actual work process, that is, to produce different objects according to different orders.

[0051] Therefore, it should be noted in this step that when the processor of the aforementioned flexible automated production line obtains the order information, the processor will immediately determine whether the order name in the current order information is consistent with the name of the item produced by the current flexible automated production line.

[0052] Step S20: If it is determined that the order name in the order information is inconsistent with the current production name, then the original neural network model weights corresponding to the production name are unloaded, and the target neural network model weights corresponding to the order name are loaded in the model weight database, the supporting information corresponding to the order name is read out, and the target neural network model weights are input into the preset neural network model.

[0053] Furthermore, in this step, it should be noted that when the processor determines that the order name in the currently obtained order information is inconsistent with the production name of the currently produced item, it indicates that the current flexible automated production line needs to change the production item, and thus the current flexible automated production line's operating program needs to be changed to meet the production requirements of the new item.

[0054] Specifically, in this step, the processor will immediately unload the weights of the original neural network model that is already running inside it, and further load the target neural network model weights corresponding to the currently obtained order name from its pre-set model weight database. At the same time, it will read the supporting information corresponding to the currently obtained order name, and then input the loaded target neural network model weights into the preset neural network model.

[0055] Correspondingly, in this step, when the processor determines that the order name in the currently obtained order information is consistent with the production name of the currently produced item, it indicates that the current flexible automated production line needs to continue producing the original item, and the original program continues to be executed.

[0056] Step S30: Obtain the entity image corresponding to the order information, and perform segmentation processing on the entity image through the preset neural network model to obtain the classification result, regression box result and mask result corresponding to the entity image;

[0057] Specifically, after changing the weights in the neural network model through the above steps, this step will further capture entity images in the current scene using the set camera, that is, obtain entity images corresponding to the current order information, and further use the neural network model loaded with the target neural network model weights to perform target instance segmentation processing on the real-time acquired entity images to obtain the classification results, regression box results and mask results corresponding to the current entity images.

[0058] Step S40: Determine whether a target object exists in the entity image based on the classification result;

[0059] Furthermore, in this step, it should be noted that after the current processor obtains the classification result, regression box result, and mask result corresponding to the currently captured entity image through the above steps, the current processor will immediately determine whether the required target object exists in the currently captured entity image based on the above classification result.

[0060] Step S50: If it is determined that the target object exists in the entity image, the spatial pose of the target object is calculated using the regression box result and the mask result. The spatial pose includes the three-dimensional coordinates and three-dimensional angles of the target object.

[0061] Finally, it should be noted in this step that if the current processor determines that there is a target object in the current entity image based on the obtained classification results, the current processor will immediately calculate the target point cloud based on the above mask results and the depth image corresponding to the current target object, and match the corresponding target pose in the preset point cloud template based on the current target point cloud; furthermore, the current processor will calculate the three-dimensional angle corresponding to the current target object based on the currently matched target pose, and the three-dimensional angle represents the three-dimensional angle of the current target object in the current shooting space.

[0062] Furthermore, in this step, it should be noted that the processor will also calculate the three-dimensional coordinates of the current target object based on its internal preset algorithm and the above regression box results, and further fuse the currently acquired three-dimensional angle and three-dimensional coordinates to finally generate the spatial pose corresponding to the current target object.

[0063] In addition, it should be noted in this embodiment that the method further includes:

[0064] If the current processor determines that the required target object does not exist in the current entity image, it will acquire the entity image corresponding to the current order information again through the camera, and then perform segmentation processing on the second acquired entity image through the preset neural network model until the target object appears.

[0065] Furthermore, in this embodiment, it should be noted that after calculating the spatial pose of the target object using the regression box results and the mask results, the method further includes:

[0066] The processor will then transmit the acquired spatial pose to the controller of the robotic arm of the composite robot, so that the controller can control the robotic arm to grasp the corresponding target object based on the spatial pose in order to perform the corresponding production task.

[0067] In operation, the system acquires order information in real time and determines whether the order name matches the production name of the object being produced. If not, it unloads the original neural network model weights corresponding to the current production name and loads the target neural network model weights corresponding to the current order name from the model weight database. It also retrieves the associated information and inputs these target neural network model weights into a preset neural network model. Further, it acquires the current entity image and segments it using the aforementioned neural network model to obtain classification results, bounding box results, and mask results. Finally, it determines whether a target object exists in the current entity image based on the classification results. If so, it calculates the spatial pose of the target object using the bounding box results and mask results. This method allows for real-time switching of neural network model weights based on the order name in the order information and segmentation of the acquired entity image using a preset neural network model. The resulting spatial pose of the target object can be easily and quickly obtained, enabling the composite robot to rapidly grasp different types of objects, significantly improving its work efficiency and making it suitable for widespread adoption.

[0068] It should be noted that the above implementation process is only to illustrate the feasibility of this application, but it does not mean that the object recognition and positioning method of this application has only the above-mentioned unique implementation process. On the contrary, as long as the object recognition and positioning method of this application can be implemented, it can be included in the feasible implementation scheme of this application.

[0069] In summary, the object recognition and localization method provided by the above embodiments of the present invention can switch the weights of the neural network model in real time according to the order name in the order information, and perform segmentation processing on the acquired entity image based on the preset neural network model. Finally, the spatial pose of the target object can be obtained simply and quickly based on the processing result, thereby enabling the composite robot to quickly grasp different types of objects, greatly improving the working efficiency of the composite robot, and is suitable for widespread promotion and use.

[0070] Please see Figure 2The figure shows an object recognition and positioning system provided in a second embodiment of the present invention, the system comprising:

[0071] The first judgment module 12 is used to determine whether the order name in the order information is consistent with the current production name when the order information is obtained;

[0072] The first execution module 22 is used to, if it is determined that the order name in the order information is inconsistent with the current production name, unload the original neural network model weights corresponding to the production name, load the target neural network model weights corresponding to the order name in the model weight database, read the supporting information corresponding to the order name, and input the target neural network model weights into the preset neural network model;

[0073] Processing module 32 is used to acquire entity images corresponding to the order information, and to segment the entity images using the preset neural network model to obtain classification results, regression box results, and mask results corresponding to the entity images;

[0074] The second judgment module 42 is used to determine whether a target object exists in the entity image based on the classification result;

[0075] The second execution module 52 is used to calculate the spatial pose of the target object by means of the regression box result and the mask result if the target object is determined to exist in the entity image. The spatial pose includes the three-dimensional coordinates and three-dimensional angles of the target object.

[0076] In the aforementioned object recognition and positioning system, the object recognition and positioning system further includes a loop module 62, which is specifically used for:

[0077] If it is determined that the target object does not exist in the entity image, a second entity image corresponding to the order information is obtained, and the second entity image is segmented using the preset neural network model until the target object appears.

[0078] In the aforementioned object recognition and positioning system, the second execution module 52 is specifically used for:

[0079] The target point cloud is calculated based on the mask result and the depth image corresponding to the target object, and the corresponding target pose is matched in the preset point cloud template based on the target point cloud.

[0080] The three-dimensional angles corresponding to the target object are calculated based on the target posture.

[0081] In the aforementioned object recognition and positioning system, the second execution module 52 is further specifically used for:

[0082] The three-dimensional coordinates of the target object are calculated based on the preset algorithm and the regression box results, and the three-dimensional angle and the three-dimensional coordinates are fused to generate the spatial pose of the target object.

[0083] In the aforementioned object recognition and positioning system, the object recognition and positioning system further includes a transmission module 72, which is specifically used for:

[0084] The spatial pose is transmitted to the controller of the robotic arm, so that the controller controls the robotic arm to grasp the target object according to the spatial pose.

[0085] The third embodiment of the present invention provides a computer, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the object recognition and positioning method provided in the first embodiment above.

[0086] The fourth embodiment of the present invention provides a readable storage medium on which a computer program is stored, which, when executed by a processor, implements the object recognition and positioning method provided in the first embodiment above.

[0087] In summary, the object recognition and positioning method, system, computer, and readable storage medium provided in the above embodiments of the present invention can switch the weights of the neural network model in real time according to the order name in the order information, and perform segmentation processing on the acquired entity image based on the preset neural network model. Finally, the spatial pose of the target object can be obtained simply and quickly based on the processing result, thereby enabling the composite robot to quickly grasp different types of objects, greatly improving the working efficiency of the composite robot, and is suitable for widespread promotion and use.

[0088] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0089] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0090] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0091] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0092] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0093] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for object recognition and localization, characterized in that, The method includes: When order information is obtained, it is determined whether the order name in the order information is consistent with the current production name; If it is determined that the order name in the order information is inconsistent with the current production name, then the original neural network model weights corresponding to the production name are unloaded, the target neural network model weights corresponding to the order name are loaded from the model weight database, the supporting information corresponding to the order name is read out, and the target neural network model weights are input into the preset neural network model. Obtain the entity image corresponding to the order information, and perform segmentation processing on the entity image through the preset neural network model to obtain the classification result, regression box result and mask result corresponding to the entity image; Based on the classification results, determine whether a target object exists in the entity image; If the target object is determined to exist in the entity image, the spatial pose of the target object is calculated using the regression box result and the mask result. The spatial pose includes the three-dimensional coordinates and three-dimensional angles of the target object. The step of calculating the spatial pose of the target object using the regression box result and the mask result includes: The target point cloud is calculated based on the mask result and the depth image corresponding to the target object, and the corresponding target pose is matched in the preset point cloud template based on the target point cloud. The three-dimensional angles corresponding to the target object are calculated based on the target posture. The three-dimensional coordinates of the target object are calculated based on the preset algorithm and the regression box results, and the three-dimensional angle and the three-dimensional coordinates are fused to generate the spatial pose of the target object. After the step of calculating the spatial pose of the target object using the regression box result and the mask result, the method further includes: The spatial pose is transmitted to the controller of the robotic arm, so that the controller controls the robotic arm to grasp the target object according to the spatial pose; If it is determined that the target object does not exist in the entity image, a second entity image corresponding to the order information is obtained, and the second entity image is segmented using the preset neural network model until the target object appears.

2. An object recognition and positioning system, characterized in that, The system includes: The first judgment module is used to determine whether the order name in the order information is consistent with the current production name when the order information is obtained; The first execution module is used to, if it is determined that the order name in the order information is inconsistent with the current production name, unload the original neural network model weights corresponding to the production name, load the target neural network model weights corresponding to the order name from the model weight database, read the supporting information corresponding to the order name, and input the target neural network model weights into the preset neural network model; The processing module is used to acquire the entity image corresponding to the order information, and to perform segmentation processing on the entity image through the preset neural network model to obtain the classification result, regression box result and mask result corresponding to the entity image; The second judgment module is used to determine whether a target object exists in the entity image based on the classification result; The second execution module is used to calculate the spatial pose of the target object by means of the regression box result and the mask result if the target object is determined to exist in the entity image. The spatial pose includes the three-dimensional coordinates and three-dimensional angles of the target object. The second execution module is specifically used for: The target point cloud is calculated based on the mask result and the depth image corresponding to the target object, and the corresponding target pose is matched in the preset point cloud template based on the target point cloud. The three-dimensional angles corresponding to the target object are calculated based on the target posture. The object recognition and positioning system further includes a loop module, which is specifically used for: If it is determined that the target object does not exist in the entity image, a second entity image corresponding to the order information is obtained, and the second entity image is segmented using the preset neural network model until the target object appears.

3. A computer, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the object recognition and localization method as described in claim 1.

4. A readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the object recognition and localization method as described in claim 1.

Images