Image Recognition Method, Apparatus, Computer System, and Readable Storage Medium

Abstract

The present disclosure provides an image recognition method, including: obtaining an image to be recognized and point cloud information of the image to be recognized; generating an image mask by using the point cloud information of the image to be recognized; obtaining feature information of an object to be recognized in a target region of the image to be recognized by using the image mask and the image to be recognized; and matching the feature information of the object to be recognized with template feature information to obtain a recognition result, where the template feature information is feature information of a target object obtained in advance. The present disclosure also provides an image recognition device, a computer system, a readable storage medium, and a computer program product.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and more particularly, to an image recognition method, apparatus, computer system, readable storage medium, and computer program product. Background Art

[0002] Automation technology refers to the process of achieving expected goals through automatic detection, information processing, analysis and judgment, and manipulation and control according to human requirements with little or no direct human participation. Automation technology has greatly improved labor productivity and is an important condition and remarkable symbol for the modernization of industry, agriculture, national defense, and science and technology. Automatic identification technology is a key technology in automation technology, which can automatically identify target objects and obtain relevant data without manual intervention.

[0003] In the process of implementing the concept of the present disclosure, the inventors found that there are at least the following problems in the related art: when using automatic identification technology to identify target objects, it is easily affected by the background environment, resulting in low recognition accuracy. Summary of the Invention

[0004] In view of this, the present disclosure provides an image recognition method, apparatus, computer system, readable storage medium, and computer program product.

[0005] One aspect of the present disclosure provides an image recognition method, including:

[0006] Obtaining an image to be recognized and the point cloud information of the image to be recognized;

[0007] Generating an image mask using the point cloud information of the image to be recognized;

[0008] Using the image mask and the image to be recognized to obtain the feature information of the object to be recognized in the target area of the image to be recognized; and

[0009] Matching the feature information of the object to be recognized with template feature information to obtain a recognition result, where the template feature information is the feature information of a target object obtained in advance.

[0010] According to an embodiment of the present disclosure, where matching the feature information of the object to be recognized with the template feature information to obtain a recognition result includes:

[0011] Determining a matching value between the object to be recognized and the target object; and

[0012] Determining that the object to be recognized is the target object when the matching value is greater than or equal to a preset matching threshold.

[0013] According to an embodiment of the present disclosure, where there are multiple objects to be recognized in the target area of the image to be recognized;

[0014] Match the feature information of the object to be recognized with the template feature information, and the recognition result includes:

[0015] Determine the matching value of each object to be recognized among multiple objects to be recognized and the target object;

[0016] Determine the objects to be recognized with a matching value greater than or equal to the preset matching threshold as candidate objects; and

[0017] Use the non-maximum suppression method to screen multiple candidate objects, and determine the object that best matches the target object from multiple candidate objects.

[0018] According to an embodiment of the present disclosure, wherein the template feature information is generated by the following operations:

[0019] Obtain the feature information of the target object, wherein the feature information includes boundary feature information and texture feature information within the boundary region;

[0020] Crop the target object according to a preset ratio to obtain the texture feature information of the target object within the boundary region;

[0021] Based on the texture feature information, determine the number of feature points in the texture feature information; and

[0022] When the number of feature points is greater than or equal to the preset feature threshold, determine the boundary feature information and texture feature information of the target object as the template feature information.

[0023] According to an embodiment of the present disclosure, wherein before obtaining the point cloud information of the image to be recognized, the above image recognition method further includes:

[0024] Receive a task for recognizing the image to be recognized, wherein the task includes the pre-recognized quantity of the target object.

[0025] According to an embodiment of the present disclosure, the above image recognition method further includes:

[0026] Based on the recognition result, determine the recognized quantity of the target object included in the image to be recognized;

[0027] Compare the recognized quantity with the pre-recognized quantity;

[0028] When the recognized quantity is greater than or equal to the pre-recognized quantity, end the execution of the task; and

[0029] When the recognized quantity is less than the pre-recognized quantity, repeat the execution of the task until the recognized quantity is greater than or equal to the pre-recognized quantity.

[0030] According to an embodiment of the present disclosure, wherein generating an image mask using the point cloud information of the image to be recognized includes:

[0031] Generate an initial image mask using the point cloud information of the image to be recognized; and

[0032] Perform morphological dilation on the initial image mask to generate an image mask.

[0033] According to an embodiment of the present disclosure, the above image recognition method further includes:

[0034] Based on the recognition result, obtain the position information of the target object; and

[0035] Determine the grasping path of the target object according to the position information of the target object.

[0036] According to an embodiment of the present disclosure, before obtaining the image to be recognized and the point cloud information of the image to be recognized, the above image recognition method further includes:

[0037] Adjust the direction of the light source irradiating the object to be recognized, or reduce the light intensity of the light source irradiating the object to be recognized; and

[0038] Perform image acquisition on the object to be recognized.

[0039] Another aspect of the present disclosure provides an image recognition device, including:

[0040] A first acquisition module for acquiring the image to be recognized and the point cloud information of the image to be recognized;

[0041] An extraction module for generating an image mask using the point cloud information of the image to be recognized;

[0042] A second acquisition module for obtaining the feature information of the object to be recognized in the target area of the image to be recognized using the image mask and the image to be recognized; and

[0043] A matching module for matching the feature information of the object to be recognized with the template feature information to obtain a recognition result, where the template feature information is the feature information of the target object obtained in advance.

[0044] Another aspect of the present disclosure provides a computer system, including:

[0045] One or more processors;

[0046] A memory for storing one or more programs,

[0047] wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the above image recognition method.

[0048] Another aspect of the present disclosure provides a computer-readable storage medium, on which executable instructions are stored, and when the instructions are executed by a processor, the processor implements the above image recognition method.

[0049] Another aspect of the present disclosure provides a computer program product, including a computer program, the computer program includes computer-executable instructions, and the instructions are used to implement the above-mentioned image recognition method when executed.

[0050] According to an embodiment of the present disclosure, since the technical means of obtaining the image to be recognized and the point cloud information of the image to be recognized; generating an image mask by using the point cloud information of the image to be recognized; obtaining the feature information of the object to be recognized in the target area of the image to be recognized by using the image mask and the image to be recognized; and matching the feature information of the object to be recognized with the template feature information to obtain a recognition result, wherein the template feature information is the feature information of the target object obtained in advance, is adopted, the background information is filtered out by using the image mask, the interference of the background environment is excluded, and the recognition accuracy of the target object is improved; therefore, at least partially overcome the technical problem that in the prior art, the automatic recognition technology is easily affected by the background environment and the recognition accuracy is low when recognizing the target object, and further achieve the technical effect of automatic recognition of the target object with high precision, speed and intelligence. Description of the Drawings

[0051] Through the following description of the embodiments of the present disclosure with reference to the drawings, the above and other objects, features and advantages of the present disclosure will become clearer. In the drawings:

[0052] Figure 1 Schematically shows an exemplary system architecture to which the image recognition method and device of the present disclosure can be applied;

[0053] Figure 2 Schematically shows a flowchart of the image recognition method according to an embodiment of the present disclosure;

[0054] Figure 3 Schematically shows an application scenario diagram of the image recognition method according to another embodiment of the present disclosure;

[0055] Figure 4 Schematically shows the image to be recognized according to an embodiment of the present disclosure;

[0056] Figure 5 Schematically shows a flowchart of picking items in a transfer box according to another embodiment of the present disclosure;

[0057] Figure 6 Schematically shows an image recognition result diagram without combining an image mask according to a comparative example of the present disclosure;

[0058] Figure 7 Schematically shows an image recognition result diagram combining an image mask according to an embodiment of the present disclosure;

[0059] Figure 8Schematically shows an RGB image corresponding to an image to be recognized according to an embodiment of the present disclosure;

[0060] Figure 9 is Figure 8 a schematic diagram of the point cloud information;

[0061] Figure 10 is Figure 9 a schematic diagram of the point cloud information generated after morphological dilation processing;

[0062] Figure 11 Schematically shows an image recognition result diagram in which all faces according to an embodiment of the present disclosure are used as templates for matching;

[0063] Figure 12 Schematically shows an image recognition result diagram in which a face with more texture feature information in the retained face is used as a template for matching according to another embodiment of the present disclosure;

[0064] Figure 13 Schematically shows an RGB image of a target object according to another embodiment of the present disclosure;

[0065] Figure 14 is Figure 13 a corresponding feature distribution diagram of the target object;

[0066] Figure 15 is Figure 14 a feature distribution diagram with a cropping border;

[0067] Figure 16 Schematically shows a block diagram of an image recognition device according to an embodiment of the present disclosure; and

[0068] Figure 17 Schematically shows a block diagram of a computer system suitable for implementing an image recognition method according to an embodiment of the present disclosure. Detailed implementation manners

[0069] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the following detailed description, for the sake of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, obviously, one or more embodiments can also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present disclosure.

[0070] The terms used herein are merely for describing specific embodiments and are not intended to limit the present disclosure. The terms "including", "comprising" and the like used herein indicate the presence of the described features, steps, operations and / or components, but do not preclude the presence or addition of one or more other features, steps, operations or components.

[0071] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0072] In cases where expressions such as "at least one of A, B, and C, etc." are used, generally, it should be interpreted according to the meaning commonly understood by those of ordinary skill in the art (for example, "a system having at least one of A, B, and C" should include, but is not limited to, a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). In cases where expressions such as "at least one of A, B, or C, etc." are used, generally, it should be interpreted according to the meaning commonly understood by those of ordinary skill in the art (for example, "a system having at least one of A, B, or C" should include, but is not limited to, a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0073] Embodiments of the present disclosure provide an image recognition method. The method includes obtaining an image to be recognized and point cloud information of the image to be recognized; generating an image mask using the point cloud information of the image to be recognized; obtaining feature information of an object to be recognized in a target region of the image to be recognized using the image mask and the image to be recognized; and matching the feature information of the object to be recognized with template feature information to obtain a recognition result, where the template feature information is feature information of a target object obtained in advance.

[0074] Figure 1 Schematically shown is an exemplary system architecture 100 to which the image recognition method and apparatus according to embodiments of the present disclosure can be applied. It should be noted that Figure 1 What is shown is only an example of a system architecture to which embodiments of the present disclosure can be applied, to help those of ordinary skill in the art understand the technical content of the present disclosure, but it does not mean that embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0075] As Figure 1As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0076] Users can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the terminal devices 101, 102, 103, such as image acquisition applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only as examples).

[0077] The terminal devices 101, 102, 103 may be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, and desktop computers, etc.

[0078] The server 105 may be a server providing various services, such as a background management server that supports the websites browsed by users using the terminal devices 101, 102, 103 (only as an example). The background management server may analyze and process data such as received user requests, etc., and feedback the processing results (such as web pages, information, or data, etc. obtained or generated according to user requests) to the terminal devices.

[0079] It should be noted that the image recognition method provided by the embodiments of the present disclosure can generally be executed by the server 105. Correspondingly, the image recognition device provided by the embodiments of the present disclosure can generally be set in the server 105. The image recognition method provided by the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and / or the server 105. Correspondingly, the image recognition device provided by the embodiments of the present disclosure can also be set in a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and / or the server 105.

[0080] For example, the image to be recognized may originally be stored in any one of the terminal devices 101, 102, or 103 (for example, the terminal device 101, but not limited to this), or stored on an external storage device and can be imported into the terminal device 101. Then, the terminal device 101 can send the image to be recognized to other terminal devices, servers, or server clusters, and the image recognition method provided by the embodiments of the present disclosure can be executed by other servers or server clusters that receive the image to be recognized.

[0081] It should be understood that Figure 1 the numbers of the terminal devices, networks, and servers in [[ ]] are merely illustrative. According to the implementation requirements, there can be any number of terminal devices, networks, and servers.

[0082] Figure 2 A flowchart of an image recognition method according to an embodiment of the present disclosure is schematically shown.

[0083] As [[ ]] Figure 2 shown, the method includes operations S210 to S240.

[0084] In operation S210, an image to be recognized and point cloud information of the image to be recognized are obtained.

[0085] According to an embodiment of the present disclosure, the image to be recognized includes an object to be recognized, and the object to be recognized may be a mobile phone with an outer packaging box or other articles with an outer packaging box.

[0086] According to an embodiment of the present disclosure, the object to be recognized may be placed in a transfer box for transportation; therefore, the image to be recognized further includes the transfer box and other non-object-to-be-recognized objects such as a cushion inside the transfer box.

[0087] According to an embodiment of the present disclosure, point cloud information can be obtained through data collection by a three-dimensional laser scanner. In the present disclosure, the point cloud information of the image to be recognized is three-dimensional point cloud information, including three-dimensional coordinates X, Y, and Z.

[0088] In operation S220, an image mask is generated using the point cloud information of the image to be recognized.

[0089] According to an embodiment of the present disclosure, the image mask can be understood as a mask that filters out background information such as the transfer box and other non-object-to-be-recognized object information such as a cushion inside the transfer box.

[0090] According to an embodiment of the present disclosure, the area where the object to be recognized in the image to be recognized is located can be defined as a target area, and the gray value at the target area in the image mask is set to 255. Non-object-to-be-recognized objects can be defined as the background, which is a non-target area in the image mask, and the gray value is set to 0.

[0091] In operation S230, using the image mask and the image to be recognized, feature information of the object to be recognized in the target area of the image to be recognized is obtained.

[0092] According to an embodiment of the present disclosure, the image mask is used on the image to be recognized to filter out the background information of the image to be recognized, improve the extraction accuracy of the feature information of the object to be recognized in the image to be recognized, and avoid extracting unfavorable or invalid information.

[0093] In operation S240, the feature information of the object to be recognized is matched with the template feature information to obtain a recognition result, where the template feature information is the feature information of the target object obtained in advance.

[0094] According to an embodiment of the present disclosure, a registration method is used to recognize the object to be recognized in the image to be recognized. More specifically, the feature information of the target object is obtained in advance, and then the feature information of the target object is used as the template feature information and compared with one or more objects to be recognized by matching to obtain the recognition result of each object to be recognized. For example, the recognition result of whether the object to be recognized is the target object is obtained.

[0095] According to an embodiment of the present disclosure, an image mask is used to filter out the background information in the image to be recognized, improve the extraction accuracy of the feature information of the object to be recognized, and avoid the interference of the environmental background area; finally, the effect of improving the recognition accuracy is achieved.

[0096] The following refers to Figures 3 to 15 and further describes the method shown in Figure 2 in combination with specific embodiments.

[0097] Figure 3 FIG. schematically shows an application scenario diagram of an image recognition method according to another embodiment of the present disclosure. Figure 4 FIG. schematically shows an image to be recognized according to an embodiment of the present disclosure.

[0098] According to an embodiment of the present disclosure, many tasks involve the application of image recognition methods, such as robotic arm de-palletizing, in-box picking of robotic arms, in-line assembly operations of robotic arms, vision-based navigation and positioning, and so on. As Figure 3 shown, it is an application scenario diagram of picking items in a transfer box of a robotic arm. Pre-shipment items are placed inside the transfer box, and the transfer box is transported to the picking station, that is, the position corresponding to the robotic arm, by a transmission device. An image of the transfer box and the pre-shipment items in the internal area of the transfer box is obtained by an image acquisition device, such as a camera taking a picture, as the image to be recognized. As Figure 4 shown, it is an image of a transfer box containing multiple mobile phone packaging boxes.

[0099] Figure 5 FIG. schematically shows a flowchart of picking items in a transfer box according to another embodiment of the present disclosure.

[0100] As Figure 5 shown, after processing the image to be recognized by using the image recognition method of the embodiment of the present disclosure, a recognition result is obtained. Based on the recognition result, the position information of the target object can be obtained; and the robotic arm determines the picking point and picking posture of the target object according to the position information of the target object, so as to finally determine the grasping path of the target object.

[0101] According to an alternative embodiment of the present disclosure, after planning the grasping path, the robotic arm grasps the target object and determines the number of target objects grasped.

[0102] According to an alternative embodiment of the present disclosure, the final number of grasped target objects can be based on the pre-identified number of target objects included in the task, where the task can be a task received for identifying the image to be identified.

[0103] According to an alternative embodiment of the present disclosure, it is possible to compare the number of target objects grasped with the pre-identified number in the task to determine whether there are still tasks to be identified, whether it is necessary to continue taking pictures, and perform processing of the image recognition method.

[0104] However, it is not limited thereto. According to other embodiments of the present disclosure, it is also possible to determine the recognized number of target objects included in the image to be identified based on the recognition result; compare the size of the recognized number and the pre-identified number; in the case where the recognized number is greater than or equal to the pre-identified number, the robotic arm can grasp the pre-identified number in the task, and the task of performing the image recognition method can be ended.

[0105] According to other embodiments of the present disclosure, in the case where the recognized number is less than the pre-identified number, after the robotic arm grasps the target objects with the recognized number, the task is repeated, that is, taking pictures, performing processing of the image recognition method, and grasping the target objects based on the recognition result; until the recognized number is greater than or equal to the pre-identified number.

[0106] According to an embodiment of the present disclosure, applying the image recognition method of the present disclosure to a scenario such as picking items in a transfer box has high automation and intelligence, liberates human labor, and improves efficiency.

[0107] Figure 6 Schematically shows an image recognition result diagram without combining an image mask according to a comparative example of the present disclosure. Figure 7 Schematically shows an image recognition result diagram combining an image mask according to an embodiment of the present disclosure. Figure 8 Schematically shows an RGB image corresponding to the image to be identified according to another embodiment of the present disclosure. Figure 9 For Figure 8 the point cloud information schematic diagram. Figure 10 For Figure 9 the point cloud information schematic diagram generated after morphological dilation processing.

[0108] According to an alternative embodiment of the present disclosure, in the scenario of picking items in a transfer box, directly matching the feature information in the image to be identified with the template feature information is easily affected by the feature information of the background environment, resulting in incorrect recognition results. Such as Figure 6As shown, it is the recognition result obtained by directly matching the feature information in the image to be recognized with the template feature information without combining the image mask. It can be seen that the edges generated by the inner wall of the transfer box and the backing plate will cause the background information to be misrecognized as the target object.

[0109] According to an embodiment of the present disclosure, as Figure 7 shown, the present disclosure proposes to directly generate an image mask using the point cloud information of the image to be recognized. The background information can be masked and the non-feature information to be recognized can be filtered out, improving the recognition accuracy.

[0110] According to an embodiment of the present disclosure, the image mask is obtained based on the point cloud information, and the point cloud information is three-dimensional data. In the present disclosure, an ROI (region of interest) can be preset. The object that appears in the ROI is defined as the target object, while the object that does not appear in the ROI is defined as the background object. For example, for an initialized image mask of an ROI without a target object, the initial values are all 0.

[0111] According to an embodiment of the present disclosure, the ROI can be set based on the actual situation. For example, the boundaries of the transfer box are preset as the length and width of the ROI; the height is preset based on the backing plate or other background objects inside the transfer box. For example, if the height of the backing plate is Min_height, then the height h of the ROI satisfies Min_height < h < the height of the transfer box edge, and min_height can be set to a certain fixed value, such as 10 mm higher than the bottom of the transfer box. When the object to be recognized is within the preset length, width, and internal height h of the ROI, this point is considered valid. As Figure 8 and Figure 9 shown, when the point cloud information appears in the ROI, it is determined that this point is valid, and the gray value of the valid point is set to 255, and then the corresponding image mask is obtained.

[0112] However, it is not limited thereto. According to an alternative embodiment of the present disclosure, as Figure 10 shown, the point cloud information image generated by the above method can also be used as the initial image mask; and the initial image mask is subjected to morphological dilation processing to generate an image mask.

[0113] According to an alternative embodiment of the present disclosure, after performing morphological dilation processing on the initial image mask once, the holes existing in the initial image mask can be filled to avoid missing valid feature information.

[0114] According to an embodiment of the present disclosure, as Figure 7 shown, the background information is masked using the image mask, and the false targets caused by background information such as the inner wall of the box and the backing plate are eliminated in the recognition result, avoiding the interference of the background information and improving the final recognition accuracy.

[0115] According to an embodiment of the present disclosure, the image recognition method of the present disclosure not only considers the interference problem of background information, but also makes corresponding considerations respectively in aspects such as the recognition process of the object to be recognized and the generation of template feature information.

[0116] According to an embodiment of the present disclosure, the image recognition method of the present disclosure can perform calculation processing based on a registration method; for example, SIFT (Scale-invariant feature transform), LINEMOD (a template matching algorithm); however, it is not limited thereto, and Shape-Based matching can also be used.

[0117] According to an optional embodiment of the present disclosure, using the Shape-Based matching algorithm, calculate the feature information of each face of the target object in advance. The feature information uses the quantized gradient direction. After obtaining the gradient direction map of the target object, it is incorporated into the template library as template feature information. For the image to be recognized, also calculate and convert it into a gradient direction map, and perform window sliding similar to the template matching technique on the gradient direction map to be recognized using the target object template feature information (i.e., the gradient direction map) in the template library, and return candidate results that exceed the preset matching threshold.

[0118] According to an optional embodiment of the present disclosure, matching the feature information of the object to be recognized with the template feature information, and the obtained recognition result may include determining the matching value between the object to be recognized and the target object; in the case where the matching value is greater than or equal to the preset matching threshold, determining the object to be recognized as the target object.

[0119] According to an embodiment of the present disclosure, using a registration method such as Shape-Based matching for the calculation of the image recognition method has a fast speed, a high recognition rate, and high robustness.

[0120] According to an optional embodiment of the present disclosure, matching the feature information of the object to be recognized with the template feature information, and the obtained recognition result may further include the following operations.

[0121] Determine the matching value between each object to be recognized among multiple objects to be recognized and the target object;

[0122] Determine the object to be recognized with a matching value greater than or equal to the preset matching threshold as a candidate object; and

[0123] Use the non-maximum suppression method to screen multiple candidate objects, and determine the object that best matches the target object from multiple candidate objects.

[0124] According to an alternative embodiment of the present disclosure, the target region of the image to be recognized includes multiple objects to be recognized; in the scenario of picking inside a box, it often occurs that the objects to be recognized are stacked on top of each other, and the outermost edges are more likely to introduce false positive recognition results. Moreover, there are cases where the feature information of multiple faces in the template feature information is used for matching and comparison as the template feature information, and the number of feature points on each face is unbalanced. In the present disclosure, NMS (non-maximum suppressing) is used, which is a simple greedy algorithm. All candidate results are sorted according to the matching value from high to low, and those results with a coincidence ratio greater than a certain coincidence threshold and a low matching value are eliminated, so as to obtain the final matching set.

[0125] According to an embodiment of the present disclosure, the use of the non-maximum suppression calculation method effectively improves the recognition accuracy of Shape-Based matching in the case of unbalanced number of features in the template feature information. However, it is not limited to this. The problem of unbalanced number of features on each face in the template feature information can also be controlled by controlling the richness of feature points in the template feature information and eliminating the template feature information with less texture or no texture.

[0126] According to an alternative embodiment of the present disclosure, the target object can be a cubic object such as a packaging box containing electronic products such as mobile phones. Among them, the packaging box has six faces, and each face has some text information or graphic information similar to product introductions.

[0127] According to an embodiment of the present disclosure, when selecting the template feature information, the feature information of all six faces can be obtained as the template feature information. However, it is not limited to this. The texture feature information of multiple obtained faces can also be compared, and the faces with no texture feature information or less texture feature information are eliminated, and only the faces with more texture feature information inside the face are retained as the template feature information.

[0128] According to other embodiments of the present disclosure, Figure 11 Without filtering, all faces are used as template feature information for matching to obtain the matching results of the objects to be recognized.

[0129] According to an embodiment of the present disclosure, Figure 12 After filtering the faces with no texture feature information or less texture feature information, the faces with more texture feature information inside the face are retained as the template for matching to obtain the matching results of the objects to be recognized.

[0130] As Figure 11 and Figure 12 shown, by filtering out the faces with less texture or no texture, only the faces with more texture feature information inside the face are retained as the template feature information, reducing the introduction of false positive results and improving the recognition accuracy.

[0131] According to an embodiment of the present disclosure, the template feature information can be generated by combining operations such as filtering feature information with little or no texture.

[0132] Figure 13 An RGB image of the target object according to an embodiment of the present disclosure is shown. Figure 14 is Figure 13 The corresponding feature distribution map of the target object in Figure 15 is Figure 14 The feature distribution map of with a cropping border.

[0133] According to an embodiment of the present disclosure, as Figure 13 shown, the feature information of the target object can be obtained by collecting the RGB image of the target object and then converting it into the corresponding grayscale image to extract the feature information of the target object in the image. The feature information of the target object in the grayscale image can be reflected by the feature points with a relatively large gradient of gray levels, but it is not limited thereto, and it can also be reflected by the feature points with relatively drastic gray level changes. In the present disclosure, the feature point only needs to be an invariant under affine transformation.

[0134] According to an embodiment of the present disclosure, as Figure 14 shown, the feature information includes boundary feature information and texture feature information within the boundary region.

[0135] According to an optional embodiment of the present disclosure, the more texture feature information within the boundary region of the target object, the more beneficial it is to improve the final recognition accuracy. According to an embodiment of the present disclosure, as Figure 15 shown, the frame line within the boundary in the figure is the new boundary scaled down based on the boundary in proportion. Cropping the target object according to the new boundary realizes the separation of the boundary feature information and the texture feature information within the boundary region, and further obtains the texture feature information of the target object within the boundary region.

[0136] According to an embodiment of the present disclosure, based on the texture feature information, determine the number of feature points in the texture feature information; in the case where the number of feature points is greater than or equal to a preset feature threshold, determine the boundary feature information and the texture feature information of the target object as the template feature information.

[0137] According to an embodiment of the present disclosure, the preset feature threshold can be appropriately adjusted according to the actual situation, and no specific limitation is made here. As long as it can be realized, when the number of feature points within the boundary region is greater than the preset feature threshold, it is considered that the surface has texture feature information and can be used as template feature information and incorporated into the template library; otherwise, it is not incorporated into the template library.

[0138] According to an embodiment of the present disclosure, the feature information of the template is filtered to remove the feature information of the surfaces with no texture or less texture, which helps to reduce the false positive results introduced by the surfaces with no texture or less texture and improve the recognition accuracy.

[0139] In addition, in addition to improving aspects such as image processing and matching algorithms, the present disclosure also proposes a problem of specular reflection that needs to be improved during the image acquisition process. In the present disclosure, for the to-be-recognized image obtained in the image recognition method, if specular reflection occurs during the image acquisition process, a specular reflection area will be displayed on the to-be-recognized image, and this specular reflection area is often misrecognized as a matched feature area. To solve this specular reflection problem, before obtaining the to-be-recognized image and the point cloud information of the to-be-recognized image, by designing the light source direction, light source intensity or installing a polarizer on the incident lens of the image acquisition device, the influence of specular reflection on image recognition is reduced in this way.

[0140] According to an optional embodiment of the present disclosure, the designed light source direction can be to adjust the direction of the light source irradiating the to-be-recognized object. For example, the light source is projected onto the surface of an object and then diffusely reflected by the object surface to the surface of the to-be-recognized object.

[0141] According to an optional embodiment of the present disclosure, reducing the light intensity of the light source irradiating the to-be-recognized object can be to block it with a light-shielding cloth in the incident direction of the light source.

[0142] According to an optional embodiment of the present disclosure, on the basis of adjusting the direction of the light source irradiating the to-be-recognized object or reducing the light source intensity, a polarizer can be installed on the incident lens to achieve a dual effect of reducing specular reflection.

[0143] In summary, the present disclosure proposes an image recognition method for identifying rigid polyhedra using Shape-Based matching, generating an image mask of an effective area using 3D point cloud information, and filtering out the surfaces with less texture in the template feature information, and only recognizing the surfaces with relatively rich internal texture feature information and other strategies, which effectively improve the recognition accuracy and reduce the misrecognition rate.

[0144] At the same time, the present disclosure also proposes methods such as changing the light source direction and installing a polarizer before image acquisition of the to-be-recognized object, solving the specular reflection problem, and further assisting in improving the recognition accuracy.

[0145] Figure 16 A block diagram of an image recognition device according to an embodiment of the present disclosure is schematically shown.

[0146] As Figure 16 shown, the image recognition device 1600 includes a first acquisition module 1610, an extraction module 1620, a second acquisition module 1630, and a matching module 1640.

[0147] A first acquisition module 1610, configured to acquire an image to be recognized and point cloud information of the image to be recognized;

[0148] An extraction module 1620, configured to generate an image mask by using the point cloud information of the image to be recognized;

[0149] A second acquisition module 1630, configured to acquire feature information of an object to be recognized in a target region of the image to be recognized by using the image mask and the image to be recognized; and

[0150] A matching module 1640, configured to match the feature information of the object to be recognized with template feature information to obtain a recognition result, where the template feature information is feature information of a target object acquired in advance.

[0151] According to an embodiment of the present disclosure, the matching module 1640 includes a first determination unit and a second determination unit.

[0152] The first determination unit is configured to determine a matching value between the object to be recognized and the target object; and

[0153] The second determination unit is configured to determine that the object to be recognized is the target object when the matching value is greater than or equal to a preset matching threshold.

[0154] According to an embodiment of the present disclosure, a plurality of objects to be recognized are included in the target region of the image to be recognized.

[0155] According to an embodiment of the present disclosure, the matching module 1640 includes a third determination unit, a fourth determination unit, and a screening unit.

[0156] The third determination unit is configured to determine a matching value between each object to be recognized among the plurality of objects to be recognized and the target object;

[0157] The fourth determination unit is configured to determine an object to be recognized with a matching value greater than or equal to a preset matching threshold as a candidate object; and

[0158] The screening unit is configured to screen the plurality of candidate objects by using a non-maximum suppression method to determine an object that best matches the target object from the plurality of candidate objects.

[0159] According to an embodiment of the present disclosure, the template feature information is generated by the following operations.

[0160] Acquire feature information of the target object, where the feature information includes boundary feature information and texture feature information within the boundary region;

[0161] Crop the target object according to a preset ratio to obtain texture feature information of the target object within the boundary region;

[0162] Determine the number of feature points in the texture feature information based on the texture feature information; and

[0163] In the case where the number of feature points is greater than or equal to a preset feature threshold, determine the boundary feature information and texture feature information of the target object as the template feature information.

[0164] According to an embodiment of the present disclosure, wherein, before obtaining the point cloud information of the image to be recognized, the image recognition device 1600 further includes a receiving module.

[0165] The receiving module is configured to receive a task for recognizing the image to be recognized, wherein the task includes a pre-recognized quantity of the target object.

[0166] According to an embodiment of the present disclosure, the image recognition device 1600 further includes a quantity determination module, a comparison module, an end task module, and a repeated execution module.

[0167] The quantity determination module is configured to determine the recognized quantity of the target object included in the image to be recognized based on the recognition result;

[0168] The comparison module is configured to compare the recognized quantity with the pre-recognized quantity;

[0169] The end task module is configured to end the execution of the task when the recognized quantity is greater than or equal to the pre-recognized quantity; and

[0170] The repeated execution module is configured to repeatedly execute the task when the recognized quantity is less than the pre-recognized quantity until the recognized quantity is greater than or equal to the pre-recognized quantity.

[0171] According to an embodiment of the present disclosure, wherein the extraction module includes a primary extraction unit and a final extraction unit.

[0172] The primary extraction unit is configured to generate an initial image mask using the point cloud information of the image to be recognized; and

[0173] The final extraction unit is configured to perform morphological dilation processing on the initial image mask to generate an image mask.

[0174] According to an embodiment of the present disclosure, the image recognition device 1600 further includes a third acquisition module and a path determination module.

[0175] The third acquisition module is configured to acquire the position information of the target object based on the recognition result; and

[0176] The path determination module is configured to determine the grasping path of the target object according to the position information of the target object.

[0177] According to an embodiment of the present disclosure, before acquiring the image to be recognized and the point cloud information of the image to be recognized, the image recognition device 1600 further includes a light source adjustment module and an image acquisition module.

[0178] The light source adjustment module is configured to adjust the direction of the light source irradiating the object to be recognized or reduce the light intensity of the light source irradiating the object to be recognized; and

[0179] The image acquisition module is configured to acquire an image of the object to be recognized.

[0180] According to an embodiment of the present disclosure, any plurality of modules, sub-modules, units, and sub-units, or at least part of the functions of any of them can be implemented in one module. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure can be split into multiple modules for implementation. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on substrate, a system on package, an application specific integrated circuit (ASIC), or can be implemented by any other reasonable way of integrating or packaging the circuit in hardware or firmware, or implemented in any one of the three implementation manners of software, hardware, and firmware, or in an appropriate combination of any several of them. Alternatively, one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding functions can be executed.

[0181] For example, any combination of the first acquisition module 1610, the extraction module 1620, the second acquisition module 1630, and the matching module 1640 may be combined and implemented in one module / unit / sub-unit, or any one of the modules / units / sub-units may be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units may be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present disclosure, at least one of the first acquisition module 1610, the extraction module 1620, the second acquisition module 1630, and the matching module 1640 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on a substrate, a system in a package, an application specific integrated circuit (ASIC), or may be implemented by any other reasonable means such as hardware or firmware that integrates or packages circuits, or may be implemented in any one of the three implementation manners of software, hardware, and firmware, or in any suitable combination of several of them. Alternatively, at least one of the first acquisition module 1610, the extraction module 1620, the second acquisition module 1630, and the matching module 1640 may be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding functions may be executed.

[0182] It should be noted that the image recognition device part in the embodiments of the present disclosure corresponds to the image recognition method part in the embodiments of the present disclosure. For the description of the image recognition device part, please refer to the image recognition method part specifically, and details will not be repeated here.

[0183] Figure 17 A block diagram of a computer system suitable for implementing the method described above according to an embodiment of the present disclosure is schematically shown. Figure 17 The computer system shown is merely an example and should not impose any limitations on the functions and usage scope of the embodiments of the present disclosure.

[0184] As Figure 17As shown, the computer system 1700 according to an embodiment of the present disclosure includes a processor 1701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1702 or a program loaded from a storage section 1708 into a random access memory (RAM) 1703. The processor 1701 may include, for example, a general-purpose microprocessor (e.g., CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), and so on. The processor 1701 may also include on-board memory for caching purposes. The processor 1701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0185] In the RAM 1703, various programs and data required for the operation of the system 1700 are stored. The processor 1701, the ROM 1702, and the RAM 1703 are connected to each other via a bus 1704. The processor 1701 performs various operations of the method flow according to an embodiment of the present disclosure by executing the programs in the ROM 1702 and / or the RAM 1703. It should be noted that the program may also be stored in one or more memories other than the ROM 1702 and the RAM 1703. The processor 1701 may also perform various operations of the method flow according to an embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0186] According to an embodiment of the present disclosure, the system 1700 may further include an input / output (I / O) interface 1705, and the input / output (I / O) interface 1705 is also connected to the bus 1704. The system 1700 may further include one or more of the following components connected to the I / O interface 1705: an input section 1706 including a keyboard, a mouse, etc.; an output section 1707 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1708 including a hard disk, etc.; and a communication section 1709 including a network interface card such as a LAN card, a modem, etc. The communication section 1709 performs communication processing via a network such as the Internet. A drive 1710 is also connected to the I / O interface 1705 as needed. A removable medium 1711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1710 as needed so that a computer program read from it can be installed into the storage section 1708 as needed.

[0187] According to an embodiment of the present disclosure, the method flow according to the embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1709, and / or installed from the removable medium 1711. When the computer program is executed by the processor 1701, the above functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the above-described system, device, apparatus, module, unit, etc. can be implemented by computer program modules.

[0188] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist separately without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0189] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include but is not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device.

[0190] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the above-described ROM 1702 and / or RAM 1703 and / or one or more memories other than ROM 1702 and RAM 1703.

[0191] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program includes program code for executing the method provided by the embodiment of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the image recognition method provided by the embodiment of the present disclosure.

[0192] When the computer program is executed by the processor 1701, the above functions defined in the system / apparatus of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the above-described system, apparatus, module, unit, etc. can be implemented by computer program modules.

[0193] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1709, and / or installed from the removable medium 1711. The program code included in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0194] According to embodiments of the present disclosure, the program code for executing the computer program provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level procedures and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include but are not limited to, for example, Java, C++, Python, the "C" language, or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., by using an Internet service provider to connect through the Internet).

[0195] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code, and the above-mentioned module, segment of a program, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, as well as the combinations of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions. Those skilled in the art can understand that the features described in various embodiments and / or claims of the present disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in various embodiments and / or claims of the present disclosure can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of the present disclosure.

[0196] The embodiments of the present disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments have been described separately above, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art can make various substitutions and modifications, and all such substitutions and modifications should fall within the scope of the present disclosure.

Claims

1. An image recognition method, comprising: Obtain the image to be recognized and the point cloud information of the image to be recognized; Generate an image mask using the point cloud information of the image to be recognized; Using the image mask and the image to be recognized, obtain the feature information of the object to be recognized in the target area of the image to be recognized; And Match the feature information of the object to be recognized with the template feature information to obtain a recognition result, where the template feature information is the feature information of the target object obtained in advance, including: when there are multiple objects to be recognized in the target area of the image to be recognized, determine the feature information of each object to be recognized among the multiple objects to be recognized and match it with the template feature information of the target object to obtain a matching value, where the template feature information is determined based on the feature information of multiple faces of the target object, and the target object is a cube object; determine the object to be recognized with a matching value greater than or equal to the preset matching threshold as a candidate object; and use the non-maximum suppression method to screen multiple candidate objects, and determine the object that best matches the target object from the multiple candidate objects; Wherein, the template feature information is generated through the following operations: Obtain the feature information of the target object, where the feature information includes boundary feature information and texture feature information within the boundary area; Crop the target object according to a preset ratio to obtain the texture feature information of the target object within the boundary area; Based on the texture feature information, determine the number of feature points in the texture feature information; and When the number of feature points is greater than or equal to the preset feature threshold, determine the boundary feature information and the texture feature information of the target object as the template feature information.

2. The method according to claim 1, wherein The matching of the feature information of the object to be recognized with the template feature information to obtain a recognition result includes: When there is one object to be recognized in the target area of the image to be recognized, determine the matching value of the object to be recognized and the target object; and When the matching value is greater than or equal to the preset matching threshold, determine the object to be recognized as the target object.

3. The method according to claim 1, wherein Before obtaining the point cloud information of the image to be recognized, the method further includes: Receive a task for recognizing the image to be recognized, where the task includes the pre-recognized quantity of the target object.

4. The method according to claim 3, further comprising: Based on the recognition result, determine the recognized quantity of the target object included in the image to be recognized; Compare the size of the recognized quantity and the pre-recognized quantity; When the recognized quantity is greater than or equal to the pre-recognized quantity, end the execution of the task; And When the recognized quantity is less than the pre-recognized quantity, repeat the execution of the task until the recognized quantity is greater than or equal to the pre-recognized quantity.

5. The method according to claim 1, wherein The generating an image mask using the point cloud information of the image to be recognized includes: Generate an initial image mask using the point cloud information of the image to be recognized; and Perform morphological dilation processing on the initial image mask to generate an image mask.

6. The method according to claim 1, further comprising: Based on the recognition result, obtain the position information of the target object; And Determine the grasping path of the target object according to the position information of the target object.

7. Before obtaining the image to be recognized and the point cloud information of the image to be recognized, the method according to claim 1 further comprises: Adjust the direction of the light source irradiating the object to be recognized, or reduce the light intensity of the light source irradiating the object to be recognized; And Perform image acquisition on the object to be recognized.

8. An image recognition device, comprising: The first acquisition module is used to acquire the image to be recognized and the point cloud information of the image to be recognized; The extraction module is used to generate an image mask by using the point cloud information of the image to be recognized; The second acquisition module is used to acquire the feature information of the object to be recognized in the target area of the image to be recognized by using the image mask and the image to be recognized; And The matching module is used to match the feature information of the object to be recognized with the template feature information to obtain a recognition result, wherein the template feature information is the feature information of the target object obtained in advance, and the target object is a cube object; The matching the feature information of the object to be recognized with the template feature information to obtain a recognition result includes: When there are multiple objects to be recognized in the target area of the image to be recognized, determine the feature information of each object to be recognized among the multiple objects to be recognized to be matched with the template feature information of the target object to obtain a matching value, wherein the template feature information is determined based on the feature information of multiple faces of the target object; Determine the object to be recognized with the matching value greater than or equal to the preset matching threshold as a candidate object; and Use the non-maximum suppression method to screen multiple candidate objects, and determine the object most matching the target object from the multiple candidate objects; Wherein, the template feature information is generated through the following operations: Acquire the feature information of the target object, wherein the feature information includes boundary feature information and texture feature information within the boundary region; Crop the target object according to a preset ratio to obtain the texture feature information of the target object within the boundary region; Based on the texture feature information, determine the number of feature points in the texture feature information; and When the number of feature points is greater than or equal to the preset feature threshold, determine the boundary feature information and the texture feature information of the target object as the template feature information.

9. A computer system, comprising: One or more processors; A memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, which when executed by a processor cause the processor to implement the method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program, the computer program comprising computer-executable instructions, the instructions being operative when executed to implement the method according to any one of claims 1 to 7.

Images