Object posture recognition method and device, computer device, storage medium and product

Abstract

The application relates to an object posture recognition method and device, computer equipment, a storage medium and a product. A scene image is acquired, a subgraph corresponding to a target object is acquired from the scene image, at least one visible surface of the target object is recognized based on the subgraph, a plurality of feature points in the subgraph are detected, matching processing is performed on the at least one visible surface and the plurality of feature points, a target data set of the target object is obtained, and finally, an object posture of the target object is obtained based on the target data set. The object posture recognition method and device can be applied to object posture recognition of various objects and can accurately obtain the object posture.

Description

Technical Field

[0001] This application relates to the field of industrial vision technology, and in particular to an object posture recognition method, apparatus, computer equipment, storage medium, and product. Background Technology

[0002] In the industrial sector, to save manpower and reduce costs, machines are increasingly being used to replace manual labor, such as using robotic arms to plug and unplug devices. Since plugs need to be inserted in a fixed orientation, recognizing the plug's orientation is crucial.

[0003] In traditional technology, a 3D camera is used to recognize the posture of the plug. The 3D camera can capture a three-dimensional image of the plug, thereby recognizing the posture of the plug in the image.

[0004] However, due to the limitations of the camera's 3D point cloud accuracy, it is impossible to model plugs that are too small or too complex, resulting in errors in pose recognition. Summary of the Invention

[0005] Therefore, it is necessary to provide an object posture recognition method, device, computer equipment, storage medium, and product that can accurately identify the posture of an object in order to address the above-mentioned technical problems.

[0006] In a first aspect, this application provides an object pose recognition method, the method comprising:

[0007] Acquire the scene image, and extract the sub-image corresponding to the target object from the scene image;

[0008] Based on subgraph recognition, at least one visible face of the target object is obtained, and multiple feature points in the subgraph are detected;

[0009] Matching is performed on at least one visible surface and multiple feature points to obtain the target dataset of the target object; the target dataset includes the target visible surface and the matching target feature points.

[0010] The object pose of the target object is obtained based on the target dataset.

[0011] In one embodiment, the scene image includes multiple objects, and the step of obtaining the sub-image corresponding to the target object from the scene image includes:

[0012] Detect all objects in the scene image and obtain the object label for each object.

[0013] Retrieve the target object label corresponding to the target object;

[0014] Match the target object label with multiple object labels to determine the area where the target object is located;

[0015] The region containing the target object is processed to obtain the sub-image corresponding to the target object.

[0016] In one embodiment, the step of detecting multiple feature points in a subgraph includes:

[0017] Detect all visible corner points in the subgraph;

[0018] All visible corner points are numbered according to the preset numbering rules, and the serial number corresponding to each identified visible corner point is output; visible corner points are feature points.

[0019] In one embodiment, the step of matching at least one visible face and multiple feature points to obtain a target dataset of the target object includes:

[0020] The matching process is performed on at least one visible face and multiple feature points to obtain at least one candidate dataset. The candidate dataset includes a visible face and a key point group corresponding to the visible face. The key point group includes a preset number of feature points.

[0021] The key point groups in each candidate dataset are validated separately, and the target dataset is selected from the candidate datasets based on the validation results.

[0022] In one embodiment, the step of verifying the keypoint groups in each candidate dataset and selecting the target dataset from the candidate datasets based on the verification results includes:

[0023] For any set of key points in a candidate dataset, examine the area of ​​the polygon formed by the set of key points. If the area is greater than the area threshold, calculate the parallelogram degree based on the set of key points.

[0024] The candidate dataset corresponding to the key point group whose parallelogram degree meets the preset conditions is output as the target dataset.

[0025] In one embodiment, the step of calculating the parallelogram degree based on the key point set includes:

[0026] Each feature point in the keypoint group is taken as the target point in turn;

[0027] For each selected target point, a parallelogram is fitted to the remaining feature points in the key point group to obtain the predicted point, and the distance between the target point and the predicted point is calculated.

[0028] The maximum value among the calculated distances is taken as the parallelogram degree.

[0029] In one embodiment, the target feature points are two-dimensional feature points, and the step of obtaining the object pose of the target object based on the target dataset includes:

[0030] Based on the target visible facet in the target dataset, select the target prior 3D feature point corresponding to the target visible facet from multiple prior 3D feature points;

[0031] The object pose is obtained by performing dimensional transformation based on the prior 3D feature points of the target and the target feature points.

[0032] In one embodiment, a dimensionality transformation is performed based on the target's prior 3D feature points and the target feature points to obtain the object pose of the target object, including:

[0033] Based on the prior 3D feature points of the target and the target feature points, a dimensional transformation is performed to obtain the candidate pose of the target object;

[0034] Calculate the position information of the center point of the quadrilateral enclosed by the target feature points;

[0035] The horizontal and vertical coordinates of the candidate poses are updated based on the position information of the center point to obtain the pose of the target object.

[0036] Secondly, this application also provides an object pose recognition device, the device comprising:

[0037] The acquisition module is used to acquire scene images and extract sub-images corresponding to target objects from the scene images;

[0038] The detection module is used to identify at least one visible face of the target object based on the sub-graph and to detect multiple feature points in the sub-graph.

[0039] The matching module is used to match at least one visible face and multiple feature points to obtain the target dataset of the target object; the target dataset includes the target visible face and the matching target feature points;

[0040] The pose module is used to obtain the pose of the target object based on the target dataset.

[0041] Thirdly, this application also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method steps of any one of the first aspects.

[0042] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method steps of any one of the first aspects.

[0043] Fifthly, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the method steps of any one of the first aspects.

[0044] The aforementioned object pose recognition method, apparatus, computer device, storage medium, and product acquire scene images and obtain sub-images corresponding to target objects from the scene images. Then, based on the sub-images, they identify at least one visible facet of the target object and detect multiple feature points in the sub-images. They perform matching processing on the at least one visible facet and multiple feature points to obtain a target dataset of the target object. Finally, based on the target dataset, they obtain the object pose of the target object. This method is applicable to the object pose recognition of various objects and accurately obtains the object pose. Attached Figure Description

[0045] Figure 1 This is an application environment diagram of an object pose recognition method in one embodiment;

[0046] Figure 2 This is a flowchart illustrating an object pose recognition method in one embodiment;

[0047] Figure 3 This is a schematic diagram of all possible visible surfaces of the plug in one embodiment;

[0048] Figure 4 This is a flowchart illustrating the steps for obtaining a subgraph in one embodiment;

[0049] Figure 5(A) is a schematic diagram of a scene image in one embodiment;

[0050] Figure 5(B) is a schematic diagram of a target object sub-graph in one embodiment;

[0051] Figure 6(A) is a schematic diagram of all visible corner points of the target object in one embodiment;

[0052] Figure 6(B) is a schematic diagram of the visible corner points of the target object and their numbers in one embodiment;

[0053] Figure 7 This is a flowchart illustrating the steps for calculating parallelogram degrees in one embodiment;

[0054] Figure 8 This is a flowchart illustrating a plug posture recognition method in one embodiment;

[0055] Figure 9(A) is a schematic diagram of the original image captured in one embodiment;

[0056] Figure 9(B) is a schematic diagram of a target plug sub-diagram in one embodiment;

[0057] Figure 10 This is a schematic diagram of a sub-diagram of the target plug in one embodiment;

[0058] Figure 11 This is a schematic diagram of the distribution of feature points in one embodiment;

[0059] Figure 12 This is a structural block diagram of an object pose recognition device in one embodiment;

[0060] Figure 13 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0062] The object pose recognition method provided in this application can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with imaging device 104 via a network. Imaging device 104 stores scene images. Terminal 102 acquires the scene images from the imaging device and obtains a sub-image corresponding to the target object from the scene images. Then, based on the sub-image, it identifies at least one visible facet of the target object and detects multiple feature points in the sub-image. Terminal 102 also performs matching processing on the at least one visible facet and multiple feature points to obtain a target dataset of the target object, and obtains the object pose of the target object based on the target dataset. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc. Imaging device 104 can be a camera.

[0063] In one embodiment, such as Figure 2 As shown, an object pose recognition method is provided, which is applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps:

[0064] S201: Obtain the scene image and extract the sub-image corresponding to the target object from the scene image.

[0065] The shooting device first takes pictures of the objects that need to be subject to object posture recognition. Before shooting, in order to avoid the environmental background from affecting the scene image, you can choose to shoot in a less complex environment, such as using a green screen as the shooting background to shoot the scene image.

[0066] The terminal uses a detection model to detect objects and their positions in a scene image. At this time, the scene image may contain multiple objects. Therefore, the terminal needs to obtain the sub-image corresponding to the target object from the scene image. Specifically, each object in the scene image has a corresponding label. The terminal obtains the target label corresponding to the target object according to the actual application requirements, matches the target label with the labels of multiple objects in the scene image, determines the area where the target object is located, and obtains the sub-image containing only the target object from the scene image.

[0067] For example, if the labels for each object are colors, and the target object is yellow, then the label named "yellow" is searched. The area where this label is located is the area where the target object is located. The terminal processes the area where the target object is located to obtain the sub-image corresponding to the target object.

[0068] The detection model can be a Region Convolutional Neural Network (R-CNN) model. The R-CNN model first searches the scene image through selective search, finding about 2,000 candidate windows. Then, the images of these 2,000 candidate windows are scaled to 227*227 pixels and then fed into the CNN. Each candidate window in the CNN extracts a feature vector, and the corresponding feature vector of each candidate window is classified and recognized to identify different objects in the scene image.

[0069] S202: Based on subgraph recognition, at least one visible face of the target object is obtained, and multiple feature points in the subgraph are detected.

[0070] In order to determine the pose of the target object, the orientation of the target object, i.e. the current visible face, must first be obtained. Here, the terminal uses a classification model to determine the visible face of the target object.

[0071] Generally, an object has six faces, such as a plug. To determine the orientation of a plug, the face with the cable and the face with the prongs do not need to be considered; the remaining four faces are the visible faces that need to be determined. The visible faces of a target object in a sub-image are usually one or two faces (at most two). The terminal identifies at least one visible face of the target object based on the sub-image. Simultaneously, the terminal uses a keypoint detection model to detect multiple feature points in the sub-image. Generally, the corner points of an object are its feature points; therefore, the terminal only needs to detect the corner points of the visible faces to obtain the target object's feature points. If there are two visible faces, there are six visible feature points in the object image. If there is only one visible face, theoretically, there should be four visible feature points in the object image. However, to ensure consistency in feature point detection, six feature points are still considered, with two of them coinciding with two points on the next visible face. See the attached image for details. Figure 3A schematic diagram of all possible visible parts of the plug.

[0072] Among them, classification models typically use logistic regression or decision trees. Logistic regression is essentially derived from linear regression and is a linear classifier, while decision trees are a non-parametric supervised learning method that can summarize decision rules from a series of data with features and labels and present these rules in a tree diagram structure to solve classification and regression problems.

[0073] S203: Perform matching processing on at least one visible face and multiple feature points to obtain the target dataset of the target object; the target dataset includes the target visible face and the matching target feature points.

[0074] The terminal performs matching processing on at least one visible face and multiple feature points to obtain a dataset of visible faces and feature points. If there is only one visible face, that face is taken as the best visible face, and the dataset of the best visible face and its corresponding multiple feature points constitutes the target dataset. If there are two visible faces, they can be divided into left and right faces, and the terminal can directly determine which visible face is left or right based on its name. For each visible face, it contains a preset number of feature points, forming a feature group. The terminal verifies each feature point group within it to obtain the best feature point group, and the target dataset of the target object is obtained based on the visible face corresponding to the best feature point group. If the left keypoint group is the best keypoint group, then the left keypoint group and the corresponding left face constitute the target dataset of the target object.

[0075] S204: Obtain the object pose of the target object based on the target dataset.

[0076] The terminal obtains the corresponding prior 3D feature points of the target based on the names of the visible faces in the target dataset. The prior 3D feature points of the target are the 3D key points of the target object in the world coordinate system, such as the corner coordinates of the target object. The feature point group obtained from the sub-image is the 2D coordinate point of the target object in the camera coordinate system. In this way, the terminal performs a position-n-point (PNP) algorithm to convert the 2D coordinate points and 3D coordinate points to determine the object pose of the target object. The pose obtained at this time is a 6D pose, which refers to a pose with 6 degrees of freedom, including 3 degrees of freedom of displacement and 3 degrees of freedom of spatial rotation.

[0077] The world coordinate system is the reference coordinate system of the environment in which the target object is located. It can describe the position of any object in the environment. The terminal takes a point in the environment as the origin and establishes a three-dimensional coordinate system in advance to obtain the 3D coordinates of the target object in the world coordinate system. These 3D coordinates include the 3D coordinates of any position of the target object. When performing PNP transformation, only the 3D coordinates of the target feature points are selected as the target prior three-dimensional feature points.

[0078] The PNP algorithm solves for the pose of the camera coordinate system relative to the world coordinate system. Given the coordinates (relative to the world coordinate system) of n 3D points and their pixel coordinates, it can estimate the camera pose by solving for the rotation matrix and translation vector from the world coordinate system to the camera coordinate system, thereby deriving the 6D pose of the object. Specifically, based on a known camera perspective projection model, the terminal uses the 3D and 2D coordinates of three points to calculate four solutions, obtaining four rotation and translation matrices. The fourth keypoint is used as a check point. The camera pose is calculated from the four solutions. Then, the fourth keypoint is reprojected, and the error is calculated. The solution with the smallest error is the optimal solution, yielding the final coordinates in the camera coordinate system, i.e., the camera pose with 6 degrees of freedom, thus obtaining the object pose of the target object.

[0079] In the above object pose recognition method, a scene image is acquired, and a sub-image corresponding to the target object is obtained from the scene image. Then, at least one visible face of the target object is identified based on the sub-image, and multiple feature points in the sub-image are detected. The at least one visible face and multiple feature points are matched to obtain the target dataset of the target object. Finally, the object pose of the target object is obtained based on the target dataset. This method is applicable to the object pose recognition of various objects and accurately obtains the object pose.

[0080] In one embodiment, such as Figure 4 As shown, the scene image includes multiple objects. The steps for obtaining the sub-image corresponding to the target object from the scene image include:

[0081] S401: Detect all objects in the scene image and obtain the object label for each object.

[0082] In this process, the shooting device first takes pictures of the objects that need to be recognized in terms of their pose. Before shooting, in order to avoid the environmental background from affecting the scene image, it is possible to choose to shoot in a less complex environment, such as using a green curtain as the shooting background to shoot the scene image. The terminal detects all objects in the scene image. At this time, the scene image also includes the object label of each object, as shown in Figure 5(A). Figure 5(A) is a scene image, which includes multiple objects of different shapes and the corresponding label of each object.

[0083] S402: Get the target object label corresponding to the target object.

[0084] The terminal obtains the target object label corresponding to the target object. For example, if the object label is a color, and the target object is yellow, then the obtained target object label is "yellow".

[0085] S403: Match the target object label with multiple object labels to determine the region where the target object is located; process the region where the target object is located to obtain the sub-image corresponding to the target object.

[0086] The terminal matches the target object label with each label in the scene image to determine the region where the target object is located, and obtains the region where the target object is located from the scene image to obtain the sub-image corresponding to the target object.

[0087] Specifically, the terminal processes the scene image by cropping the area containing the target object from the scene image to obtain a sub-image corresponding to the target object. As shown in Figure 5(A), the object label is a number; assuming the target object label is 3, the terminal matches the target object label with each label in the scene image to determine the area containing the target object, and then crops the area containing the target object from the scene image to obtain a sub-image corresponding to the target object. The cropped sub-image is shown in Figure 5(B).

[0088] In this embodiment, a detection model detects all objects in the scene image and obtains the object label for each object. Then, the target object label corresponding to the target object is obtained. The target object label is matched with multiple object labels to determine the region where the target object is located. The region where the target object is located is processed to obtain a sub-image corresponding to the target object. This can accurately obtain a sub-image containing only the target object, so that when performing visible corner detection in the subsequent process, it can directly focus on the sub-image of the target object without being affected by other objects in the scene image, thus improving the accuracy of visible corner detection.

[0089] In one embodiment, the step of detecting multiple feature points in a sub-image includes: detecting all visible corner points in the sub-image; numbering all visible corner points according to a preset numbering rule, and outputting the sequence number corresponding to each identified visible corner point; the visible corner points are feature points.

[0090] In this process, the terminal detects all visible corner points in the sub-image using a keypoint detection model. Typically, there are six visible corner points in an object image. The terminal uses a preset numbering rule, such as clockwise, to sequentially number each visible corner point, outputting the corresponding sequence number for each detected corner point: 1->2->3->4->5->6. If there is only one visible corner point, then numbers 2 and 3 in the sub-image are the same corner point, and numbers 4 and 5 overlap, also being the same corner point. Here, visible corner points are actually multiple feature points in the sub-image. Taking the object in Figure 5(B) as an example, the detected visible corner points are shown in Figure 6(A). Each black circle in Figure 6(A) represents a visible corner point. The image obtained by sequentially numbering each visible corner point clockwise is shown in Figure 6(B). Keypoints are essentially points in an image that represent specific parts of an object. Common keypoint detection models include object detection models. Object detection models identify target objects as boxes aligned with the center axis of the image, list an almost exhaustive list of potential object locations, and classify each location as a visible corner point of the target object at different locations.

[0091] In this embodiment, by detecting all visible corner points in the sub-image and numbering them according to a preset numbering rule, the serial number corresponding to each identified visible corner point is output, thus accurately identifying all feature points in the sub-image.

[0092] In one embodiment, the step of matching at least one visible face and multiple feature points to obtain a target dataset of a target object includes: matching at least one visible face and multiple feature points to obtain at least one set of candidate datasets, wherein the candidate datasets include a visible face and a key point group corresponding to the visible face, and the key point group includes a preset number of feature points; verifying the key point group in each set of candidate datasets respectively, and selecting the target dataset from the candidate datasets based on the verification results.

[0093] The terminal performs matching processing on at least one visible face and multiple feature points to obtain a dataset of visible faces and feature points. If there is only one visible face, that face is taken as the best visible face, and the dataset of the best visible face and its corresponding multiple feature points is the target dataset. If there are two visible faces, they can be divided into left and right faces. The terminal can directly determine which visible face is left or right based on its name. Each visible face contains four feature points, forming a feature group. For each visible face, the feature point group within it is verified to obtain the best feature point group. The target dataset of the target object is obtained based on the visible face corresponding to the best feature point group. For example, in Figure 6(B), the visible faces include the left visible face and the right visible face. The feature group of the left visible face includes 1, 2, 5, and 6, and the feature group of the right visible face includes 2, 3, 4, and 5. The terminal verifies the feature groups of the left and right visible faces respectively. If the best feature point group is 2, 3, 4, and 5, then the target dataset includes the right visible face and the feature group 2, 3, 4, and 5.

[0094] In this embodiment, by matching at least one visible face and multiple feature points, at least one set of candidate datasets is obtained, and the key point groups in each set of candidate datasets are verified respectively. Based on the verification results, the target dataset is selected from the candidate datasets, which can select the most accurate target dataset and ensure the accuracy of object pose recognition.

[0095] In one embodiment, the step of verifying the key point groups in each candidate dataset and selecting the target dataset from the candidate dataset based on the verification results includes: for any key point group in the candidate dataset, verifying the area of ​​the polygon formed by the key point group; if the area is greater than the area threshold, calculating the parallelogram degree based on the key point group; and outputting the candidate dataset corresponding to the key point group whose parallelogram degree meets the preset condition as the target dataset.

[0096] The area threshold can be preset manually or set according to the object type; no single limitation is imposed here. Parallelogram degree indicates whether the polygon formed by four points in a feature point group in the target dataset conforms to the definition of a parallelogram. The larger the parallelogram degree value, the less the polygon resembles a parallelogram. The terminal first calculates the area of ​​the polygon formed by the keypoint groups in at least one candidate dataset. If the area is greater than a certain area threshold, the verification is successful. Then, the parallelogram degree is calculated based on the keypoint groups, and the candidate dataset corresponding to the keypoint groups whose parallelogram degree meets the preset conditions is output as the target dataset. As shown in Figure 6(B), the candidate dataset includes the left dataset and the right dataset, which respectively include the corresponding visible surfaces and feature point groups. For the keypoint groups in the left visible surface, the area of ​​the polygon formed by feature point groups 1, 2, 5, and 6 is first calculated. If the area is greater than a certain area threshold, the parallelogram degree of each point in feature point group 1, 2, 5, and 6 is calculated, and the largest parallelogram degree value is taken as the parallelogram degree of this feature point group. Similarly, for the key point group visible on the right, the terminal calculates the parallelogram degree of feature point group 2, 3, 4, and 5, compares the values ​​of the two, and obtains the feature point group with the smaller parallelogram degree value. Then, the terminal outputs the candidate dataset corresponding to this feature point group as the target dataset.

[0097] In this embodiment, for any set of key points in the candidate dataset, the area of ​​the polygon formed by the set of key points is examined. If the area is greater than the area threshold, the parallelogram degree is calculated based on the set of key points. The candidate dataset corresponding to the set of key points whose parallelogram degree meets the preset condition is output as the target dataset. This can accurately obtain the target dataset and ensure the accuracy of object pose recognition.

[0098] In one embodiment, such as Figure 7 As shown, the steps for calculating the parallelogram degree based on the key point set include:

[0099] S701: Take any feature point in the key point group as the target point in sequence.

[0100] The terminal selects one key point from the four key points in the key point group as the target point. The target point can be selected arbitrarily, as shown in Figure 6(B), where the target point can be either feature 1 or feature point 2.

[0101] S702: For each selected target point, perform parallelogram fitting on the remaining feature points in the keypoint group to obtain the predicted point, and calculate the distance between the target point and the predicted point.

[0102] In this process, after selecting a target point, the terminal performs parallelogram fitting on the other feature points to obtain prediction points, resulting in four prediction points. The terminal then calculates the distance between each target point and its corresponding prediction point. As shown in Figure 6(B), assuming feature point 1 is selected as the key point, the terminal performs parallelogram fitting on the remaining three key points based on feature point 1 to obtain prediction points and calculates the distance between each target point and its corresponding prediction point. Then, feature point 2 is selected as the target point, and the fitting process is repeated, with the corresponding distance calculated.

[0103] S703: Use the maximum value among the calculated distances as the parallelogram degree.

[0104] The terminal calculates four distance values, and the maximum value among these distance values ​​is taken as the parallelogram degree.

[0105] In this embodiment, by taking any feature point in the key point group as the target point in turn, and for each selected target point, fitting a parallelogram to the remaining feature points in the key point group to obtain the predicted point, and calculating the distance between the target point and the predicted point, the maximum value among the calculated distances is taken as the parallelogram degree, which can accurately obtain the parallelogram degree, thereby ensuring the accuracy of object pose recognition.

[0106] In one embodiment, the prior 3D feature points are 3D feature points and the target feature points are 2D feature points. Obtaining the object pose of the target object based on the target dataset includes: selecting the target prior 3D feature points corresponding to the target visible surfaces from multiple prior 3D feature points according to the target visible surfaces in the target dataset; and performing dimensional transformation based on the target prior 3D feature points and the target feature points to obtain the object pose of the target object.

[0107] In this process, the terminal selects the target prior 3D feature points corresponding to the target visible face from multiple prior 3D feature points based on the name of the target visible face in the target dataset. The target prior 3D feature points are the pre-set 3D key points of the target object, while the feature point group obtained from the sub-image is the 2D coordinate point of the target object. The terminal converts the 2D coordinate points and 3D coordinate points using the PNP algorithm to obtain the object pose of the target object.

[0108] In this embodiment, by selecting the target prior 3D feature points corresponding to the target visible surfaces from multiple prior 3D feature points based on the target visible surfaces in the target dataset, and performing dimensional transformation based on the target prior 3D feature points and the target feature points, the object pose of the target object can be obtained, which can accurately identify the object pose of the target object.

[0109] In one embodiment, the step of performing a dimensional transformation based on the target prior 3D feature points and the target feature points to obtain the object pose of the target object includes: performing a dimensional transformation based on the target prior 3D feature points and the target feature points to obtain the candidate pose of the target object; calculating the position information of the center point of the quadrilateral enclosed by the target feature points; and updating the horizontal axis coordinate value and the vertical axis coordinate value in the candidate pose based on the position information of the center point to obtain the object pose of the target object.

[0110] The terminal performs dimensional transformation based on the target's prior 3D feature points and the target's feature points to obtain the object's pose as a candidate pose. In order to obtain a more accurate object pose, the terminal calculates the position information of the center point of the quadrilateral enclosed by the target feature points and represents the x and y values ​​in the candidate pose using the coordinates of the center point.

[0111] In this embodiment, by calculating the position information of the center point of the quadrilateral enclosed by the target feature points, and updating the horizontal and vertical coordinate values ​​of the object pose based on the position information of the center point, the final object pose of the target object is obtained, which can accurately identify the object pose.

[0112] In one embodiment, such as Figure 8 As shown, a plug pose recognition method based on 2D vision is provided. Taking a plug as an example, the method includes the following steps:

[0113] 1. In the current environment, the terminal acquires the plug image, as shown in Figure 9(A). Figure 9(A) is the original image of the plug, which includes various plugs and their corresponding plug labels.

[0114] 2. The terminal locates the target object from the scene image based on the plug label, and processes the area where the target plug is located from the plug image to obtain the sub-image corresponding to the target plug, as shown in Figure 9(B).

[0115] 3. The terminal determines at least one visible surface of the current plug, and all possible visible surfaces of the plug are as described above. Figure 3 As shown, the visible surface of the actual determined plug is as follows. Figure 10 As shown.

[0116] 4. The terminal identifies all visible feature points of the current plug and numbers them. Point number 1 must be the top-left corner of the plug. Then, the feature points are sorted clockwise. Figure 10 The black circle in the middle represents...

[0117] 5. The terminal matches visible faces with visible feature points and outputs the best 4 visible feature points and their corresponding names: Taking an input with 2 visible faces as an example, the left and right faces are obtained. The names of the visible faces directly determine which face is left or right. According to the feature number, the 6 visible feature points are divided into a left feature point group and a right feature point group. First, the area of ​​the polygon formed by the feature point group (with 4 keypoints) is checked. If the area is less than a certain threshold, the current group is considered unsuitable. If neither feature point group is suitable, an error is directly reported, indicating that the plug cannot obtain 6D pose. The "parallelogram degree" of the current group is checked, and the 4 feature points that best fit the parallelogram are selected as the best feature points. If the input has 2 visible faces, the corresponding visible face is selected as the best visible face based on whether the best feature points are from the left or right keypoint group. Figure 11 As shown.

[0118] 6. The terminal uses the four visible keypoints as 2D coordinate points for PNP. Based on the name of the best visible face, it selects four 3D coordinate points for that face from the prior 3D keypoints. The 2D and 3D coordinate points are then converted using the PNP algorithm, outputting the current plug's 6D pose. To obtain a better 6D pose, the center point of the quadrilateral formed by the four visible keypoints needs to be calculated. The x and y values ​​in the 6D pose obtained from PNP are then replaced with the center point coordinates, such as... Figure 11 As shown.

[0119] In this embodiment, a 2D camera is used directly to perform attitude recognition on the plug. This does not require restricting the plug's degrees of freedom and can be used for plugs of various sizes, thus improving the accuracy of plug attitude recognition.

[0120] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0121] Based on the same inventive concept, this application also provides an object posture recognition device for implementing the object posture recognition method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more object posture recognition device embodiments provided below can be found in the limitations of the object posture recognition method described above, and will not be repeated here.

[0122] In one embodiment, such as Figure 12 As shown, an object pose recognition device is provided, including: an acquisition module 10, a detection module 20, a matching module 30, and a pose module 40, wherein:

[0123] The acquisition module 10 is used to acquire scene images and extract sub-images corresponding to target objects from the scene images.

[0124] The detection module 20 is used to identify at least one visible face of the target object based on the sub-graph and to detect multiple feature points in the sub-graph.

[0125] The matching module 30 is used to perform matching processing on at least one visible face and multiple feature points to obtain the target dataset of the target object; the target dataset includes the target visible face and the matching target feature points.

[0126] The pose module 40 is used to obtain the object pose of the target object based on the target dataset.

[0127] In one embodiment, the acquisition module 10 includes: a detection unit, an acquisition unit, and a processing unit, wherein:

[0128] The detection unit is used to detect all objects in the scene image and obtain the object label of each object.

[0129] The acquisition unit is used to acquire the target object label corresponding to the target object.

[0130] The processing unit is used to match the target object label with multiple object labels to determine the region where the target object is located; and to process the region where the target object is located to obtain the sub-image corresponding to the target object.

[0131] In one embodiment, the detection module 20 includes: a detection unit and a numbering unit, wherein:

[0132] The detection unit is used to detect all visible corner points in the sub-image and treat each visible corner point as a feature point.

[0133] The numbering unit is used to number all visible corner points according to a preset numbering rule and output the serial number corresponding to each identified visible corner point; the visible corner points are feature points.

[0134] In one embodiment, the matching module 30 includes a matching unit and a verification unit, wherein:

[0135] A matching unit is used to perform matching processing on at least one visible face and multiple feature points to obtain at least one set of candidate datasets. The candidate datasets include a visible face and a set of key points corresponding to the visible face. The set of key points includes a preset number of feature points.

[0136] The verification unit is used to verify the key point groups in each candidate dataset separately, and to select the target dataset from the candidate datasets based on the verification results.

[0137] In one embodiment, the verification unit includes a calculation unit and an output unit, wherein:

[0138] The calculation unit is used to examine the area of ​​the polygon formed by the key point group in any candidate dataset. If the area is greater than the area threshold, the parallelogram degree is calculated based on the key point group.

[0139] The output unit is used to output the candidate dataset corresponding to the key point group whose parallelogram degree meets the preset conditions as the target dataset.

[0140] In one embodiment, the determination unit, the fitting unit, and the acquisition unit are defined, wherein:

[0141] The determination unit is used to sequentially select any feature point in the key point group as the target point.

[0142] The fitting unit is used to fit a parallelogram to the remaining feature points in the keypoint group for each selected target point to obtain the predicted point, and to calculate the distance between the target point and the predicted point.

[0143] The acquisition unit is used to take the maximum value among the calculated distances as the parallelogram degree.

[0144] In one embodiment, the attitude module 40 includes: a selection unit and a conversion unit, wherein:

[0145] The selection unit is used to select the target prior 3D feature point corresponding to the target visible surface from multiple prior 3D feature points based on the target visible surface in the target dataset.

[0146] The transformation unit is used to perform dimensional transformation based on the prior 3D feature points of the target and the target feature points to obtain the object pose of the target object.

[0147] In one embodiment, the transformation unit includes: a transformation subunit, a calculation subunit, and an update subunit, wherein:

[0148] The transformation subunit is used to perform dimensional transformation based on the target's prior 3D feature points and the target's feature points to obtain the candidate pose of the target object.

[0149] The computational sub-unit is used to calculate the position information of the center point of the quadrilateral enclosed by the target feature points.

[0150] The update sub-unit is used to update the horizontal and vertical coordinate values ​​in the candidate pose based on the position information of the center point, so as to obtain the object pose of the target object.

[0151] Each module in the aforementioned object pose recognition device can be implemented entirely or partially through software, hardware, and their datasets. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0152] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 13 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements an object posture recognition method. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0153] Those skilled in the art will understand that Figure 13 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0154] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: acquiring a scene image and acquiring a sub-image corresponding to a target object from the scene image; identifying at least one visible facet of the target object based on the sub-image and detecting multiple feature points in the sub-image; performing matching processing on the at least one visible facet and the multiple feature points to obtain a target dataset of the target object; the target dataset includes the target visible facet and the matching target feature points; and obtaining the object pose of the target object based on the target dataset.

[0155] In one embodiment, when the processor executes a computer program, the scene image involved includes multiple objects. Obtaining a sub-image corresponding to a target object from the scene image includes: detecting all objects in the scene image and obtaining an object label for each of the objects; obtaining a target object label corresponding to the target object; matching the target object label with multiple object labels to determine the region where the target object is located; and processing the region where the target object is located to obtain a sub-image corresponding to the target object.

[0156] In one embodiment, the detection of multiple feature points in a sub-graph when the processor executes a computer program includes: detecting all visible corner points in the sub-graph; numbering all visible corner points according to a preset numbering rule, and outputting the sequence number corresponding to each identified visible corner point; the visible corner points are feature points.

[0157] In one embodiment, the matching process involving at least one visible face and multiple feature points to obtain a target dataset of a target object when the processor executes a computer program includes: matching at least one visible face and multiple feature points to obtain at least one set of candidate datasets, wherein the candidate datasets include a visible face and a key point group corresponding to the visible face, and the key point group includes a preset number of feature points; verifying the key point group in each set of candidate datasets respectively, and selecting the target dataset from the candidate datasets based on the verification results.

[0158] In one embodiment, when the processor executes the computer program, it verifies the key point groups in each candidate dataset and selects the target dataset from the candidate dataset based on the verification results. This includes: for any key point group in the candidate dataset, verifying the area of ​​the polygon formed by the key point group; if the area is greater than an area threshold, calculating the parallelogram degree based on the key point group; and outputting the candidate dataset corresponding to the key point group whose parallelogram degree meets the preset conditions as the target dataset.

[0159] In one embodiment, the calculation of parallelogram degree based on a set of key points when the processor executes a computer program includes: sequentially taking any feature point in the set of key points as a target point; for each selected target point, fitting a parallelogram to the remaining feature points in the set of key points to obtain a predicted point, and calculating the distance between the target point and the predicted point; and taking the maximum value among the calculated distances as the parallelogram degree.

[0160] In one embodiment, when the processor executes the computer program, the target feature points involved are two-dimensional feature points. Obtaining the object pose of the target object based on the target dataset includes: selecting the target prior three-dimensional feature points corresponding to the target visible surface from multiple prior three-dimensional feature points according to the target visible surface in the target dataset; and performing dimensional transformation based on the target prior three-dimensional feature points and the target feature points to obtain the object pose of the target object.

[0161] In one embodiment, the process of a processor executing a computer program involving dimensional transformation based on prior 3D feature points of the target and target feature points to obtain the object pose of the target object includes: performing dimensional transformation based on prior 3D feature points of the target and target feature points to obtain candidate poses of the target object; calculating the position information of the center point of the quadrilateral enclosed by the target feature points; and updating the horizontal and vertical coordinate values ​​in the candidate poses based on the position information of the center point to obtain the object pose of the target object.

[0162] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon. When executed by a processor, the computer program performs the following steps: acquiring a scene image and acquiring a sub-image corresponding to a target object from the scene image; identifying at least one visible facet of the target object based on the sub-image and detecting multiple feature points in the sub-image; performing matching processing on the at least one visible facet and the multiple feature points to obtain a target dataset of the target object; the target dataset includes the target visible facet and the matching target feature points; and obtaining the object pose of the target object based on the target dataset.

[0163] In one embodiment, when a computer program is executed by a processor, the scene image involved includes multiple objects. Obtaining a sub-image corresponding to a target object from the scene image includes: detecting all objects in the scene image and obtaining an object label for each of the objects; obtaining a target object label corresponding to the target object; matching the target object label with multiple object labels to determine the region where the target object is located; and processing the region where the target object is located to obtain a sub-image corresponding to the target object.

[0164] In one embodiment, when a computer program is executed by a processor, the detection of multiple feature points in a sub-graph includes: detecting all visible corner points in the sub-graph; numbering all visible corner points according to a preset numbering rule, and outputting the sequence number corresponding to each identified visible corner point; the visible corner points are feature points.

[0165] In one embodiment, when a computer program is executed by a processor, the matching process involving at least one visible face and multiple feature points to obtain a target dataset of a target object includes: matching at least one visible face and multiple feature points to obtain at least one set of candidate datasets, each candidate dataset including a visible face and a set of key points corresponding to the visible face, the key point set including a preset number of feature points; verifying the key point set in each set of candidate datasets respectively, and selecting the target dataset from the candidate datasets based on the verification results.

[0166] In one embodiment, when the computer program is executed by the processor, it involves verifying the key point groups in each candidate dataset and selecting the target dataset from the candidate dataset based on the verification results. This includes: for any key point group in the candidate dataset, verifying the area of ​​the polygon formed by the key point group; if the area is greater than an area threshold, calculating the parallelogram degree based on the key point group; and outputting the candidate dataset corresponding to the key point group whose parallelogram degree meets the preset conditions as the target dataset.

[0167] In one embodiment, the calculation of parallelogram degree based on a set of key points when the computer program is executed by a processor includes: sequentially taking any feature point in the set of key points as a target point; for each selected target point, fitting a parallelogram to the remaining feature points in the set of key points to obtain a predicted point, and calculating the distance between the target point and the predicted point; and taking the maximum value among the calculated distances as the parallelogram degree.

[0168] In one embodiment, when the computer program is executed by the processor, the target feature points involved are two-dimensional feature points. Obtaining the object pose of the target object based on the target dataset includes: selecting the target prior three-dimensional feature points corresponding to the target visible surface from multiple prior three-dimensional feature points according to the target visible surface in the target dataset; and performing dimensional transformation based on the target prior three-dimensional feature points and the target feature points to obtain the object pose of the target object.

[0169] In one embodiment, when a computer program is executed by a processor, the dimensional transformation based on prior 3D feature points of the target and the target feature points to obtain the object pose of the target object includes: performing dimensional transformation based on prior 3D feature points of the target and the target feature points to obtain candidate poses of the target object; calculating the position information of the center point of the quadrilateral enclosed by the target feature points; and updating the horizontal and vertical coordinate values ​​in the candidate poses based on the position information of the center point to obtain the object pose of the target object.

[0170] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: acquiring a scene image and acquiring a sub-image corresponding to a target object from the scene image; identifying at least one visible facet of the target object based on the sub-image and detecting multiple feature points in the sub-image; performing matching processing on the at least one visible facet and the multiple feature points to obtain a target dataset of the target object; the target dataset includes the target visible facet and the matching target feature points; and obtaining the object pose of the target object based on the target dataset.

[0171] In one embodiment, when a computer program is executed by a processor, the scene image involved includes multiple objects. Obtaining a sub-image corresponding to a target object from the scene image includes: detecting all objects in the scene image and obtaining an object label for each of the objects; obtaining a target object label corresponding to the target object; matching the target object label with multiple object labels to determine the region where the target object is located; and processing the region where the target object is located to obtain a sub-image corresponding to the target object.

[0172] In one embodiment, when a computer program is executed by a processor, the detection of multiple feature points in a sub-graph includes: detecting all visible corner points in the sub-graph; numbering all visible corner points according to a preset numbering rule, and outputting the sequence number corresponding to each identified visible corner point; the visible corner points are feature points.

[0173] In one embodiment, when a computer program is executed by a processor, the matching process involving at least one visible face and multiple feature points to obtain a target dataset of a target object includes: matching at least one visible face and multiple feature points to obtain at least one set of candidate datasets, each candidate dataset including a visible face and a set of key points corresponding to the visible face, the key point set including a preset number of feature points; verifying the key point set in each set of candidate datasets respectively, and selecting the target dataset from the candidate datasets based on the verification results.

[0174] In one embodiment, when the computer program is executed by the processor, it involves verifying the key point groups in each candidate dataset and selecting the target dataset from the candidate dataset based on the verification results. This includes: for any key point group in the candidate dataset, verifying the area of ​​the polygon formed by the key point group; if the area is greater than an area threshold, calculating the parallelogram degree based on the key point group; and outputting the candidate dataset corresponding to the key point group whose parallelogram degree meets the preset conditions as the target dataset.

[0175] In one embodiment, the calculation of parallelogram degree based on a set of key points when the computer program is executed by a processor includes: sequentially taking any feature point in the set of key points as a target point; for each selected target point, fitting a parallelogram to the remaining feature points in the set of key points to obtain a predicted point, and calculating the distance between the target point and the predicted point; and taking the maximum value among the calculated distances as the parallelogram degree.

[0176] In one embodiment, when the computer program is executed by the processor, the target feature points involved are two-dimensional feature points. Obtaining the object pose of the target object based on the target dataset includes: selecting the target prior three-dimensional feature points corresponding to the target visible surface from multiple prior three-dimensional feature points according to the target visible surface in the target dataset; and performing dimensional transformation based on the target prior three-dimensional feature points and the target feature points to obtain the object pose of the target object.

[0177] In one embodiment, when a computer program is executed by a processor, the dimensional transformation based on prior 3D feature points of the target and the target feature points to obtain the object pose of the target object includes: performing dimensional transformation based on prior 3D feature points of the target and the target feature points to obtain candidate poses of the target object; calculating the position information of the center point of the quadrilateral enclosed by the target feature points; and updating the horizontal and vertical coordinate values ​​in the candidate poses based on the position information of the center point to obtain the object pose of the target object.

[0178] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0179] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0180] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for object pose recognition, characterized in that, The method includes: Acquire a scene image, and obtain a sub-image corresponding to the target object from the scene image; wherein, the target object includes a plug; Based on the sub-image, one or two visible faces of the target object are identified, and all visible corner points in the sub-image are detected; all visible corner points are numbered according to a preset numbering rule, and the serial number corresponding to each identified visible corner point is output, wherein the visible corner point is a feature point; The target dataset of the target object is obtained by matching one or two visible surfaces and the plurality of feature points; the target dataset includes the target visible surfaces and the matching target feature points. The object pose of the target object is obtained based on the target dataset.

2. The method according to claim 1, characterized in that, The scene image includes multiple objects, and obtaining the sub-image corresponding to the target object from the scene image includes: Detect all objects in the scene image and obtain the object label for each of the objects; Get the target object label corresponding to the target object; The target object tag is matched with multiple object tags to determine the area where the target object is located; The region where the target object is located is processed to obtain a sub-image corresponding to the target object.

3. The method according to claim 1, characterized in that, The matching process of the at least one visible surface and the plurality of feature points to obtain the target dataset of the target object includes: The at least one visible face and the plurality of feature points are matched to obtain at least one candidate dataset, wherein the candidate dataset includes a visible face and a key point group corresponding to the visible face, and the key point group includes a preset number of feature points; The key point groups in each candidate dataset are verified separately, and the target dataset is selected from the candidate datasets based on the verification results.

4. The method according to claim 3, characterized in that, The step of verifying the keypoint groups in each candidate dataset and selecting the target dataset from the candidate datasets based on the verification results includes: For any set of key points in a candidate dataset, examine the area of ​​the polygon formed by the set of key points. If the area is greater than the area threshold, calculate the parallelogram degree based on the set of key points. The candidate dataset corresponding to the key point group whose parallelogram degree meets the preset conditions is output as the target dataset.

5. The method according to claim 4, characterized in that, The calculation of parallelogram degree based on the key point set includes: Each feature point in the key point group is taken as the target point in turn; For each selected target point, a parallelogram is fitted to the remaining feature points in the key point group to obtain a predicted point, and the distance between the target point and the predicted point is calculated. The maximum value among the calculated distances is taken as the parallelogram degree.

6. The method according to claim 1, characterized in that, The target feature points are two-dimensional feature points, and obtaining the object pose of the target object based on the target dataset includes: Based on the target visible surface in the target dataset, select the target prior 3D feature point corresponding to the target visible surface from multiple prior 3D feature points; The object pose of the target object is obtained by performing dimensional transformation based on the prior 3D feature points of the target and the target feature points.

7. The method according to claim 6, characterized in that, The step of performing dimensional transformation based on the prior 3D feature points of the target and the target feature points to obtain the object pose of the target object includes: Based on the prior 3D feature points of the target and the target feature points, a dimensional transformation is performed to obtain the candidate pose of the target object; Calculate the position information of the center point of the quadrilateral enclosed by the target feature points; The horizontal and vertical coordinates of the candidate poses are updated based on the position information of the center point to obtain the pose of the target object.

8. An object posture recognition device, characterized in that, The device includes: An acquisition module is used to acquire a scene image and acquire a sub-image corresponding to a target object from the scene image; wherein, the target object includes a plug; The detection module is used to identify one or two visible faces of the target object based on the sub-image, and to detect all visible corner points in the sub-image; to number all visible corner points according to a preset numbering rule, and to output the sequence number corresponding to each identified visible corner point, wherein the visible corner points are feature points; A matching module is used to perform matching processing on one or two visible surfaces and the plurality of feature points to obtain a target dataset of the target object; the target dataset includes the target visible surfaces and the matching target feature points; The pose module is used to obtain the pose of the target object based on the target dataset.

9. The apparatus according to claim 8, characterized in that, The acquisition module includes a detection unit, an acquisition unit, and a processing unit. The detection unit is used to detect all objects in the scene image and obtain the object label of each object among all objects; The acquisition unit is used to acquire the target object label corresponding to the target object. The processing unit is used to match the target object tag with multiple object tags to determine the region where the target object is located; The region where the target object is located is processed to obtain a sub-image corresponding to the target object.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images