A deformable object classification method and apparatus based on optical flow change amount and continuous perception setting

Abstract

The application discloses a deformable object classification method and device based on optical flow change amount and continuous perception setting, comprising the following steps: acquiring a depth image sequence and a video frame sequence of a deformable object in a deformation process; using a depth convolution network to calculate a classification result of the deformable object in each depth image; calculating an optical flow change amount between adjacent video frames according to the video frame sequence; determining the effectiveness of the optical flow change amount between the adjacent video frames according to the classification results of adjacent two depth images; calculating an optical flow change amount cumulative value of each object category according to the effectiveness of the optical flow change amount; and screening an object category with the largest optical flow change amount cumulative value as a final classification result determined in the deformation process. The method and device can improve the robustness and accuracy of object classification without providing additional sensor data or manual auxiliary information.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deformable object classification, and in particular to a deformable object classification method and device based on optical flow change and continuous perception setting. BACKGROUND

[0002] Deformable objects have very important applications in various fields of robot operation, including operations on human tissues in robotic surgery, various tasks such as making dumplings, rolling dough, making sushi, etc. in home interaction, and assembly of deformable objects in industry. Deformable object manipulation is the basis for practical problems such as large object automatic cleaning, object folding, and precise grasping.

[0003] Deformable objects are mainly divided into three categories: one-dimensional deformable objects represented by ropes, two-dimensional deformable objects represented by cloth, and three-dimensional deformable objects represented by dough and plasticine.

[0004] As a subtask of deformable object classification, the object class needs to be identified according to the collected images, depth, etc. during the process of lifting the object from the table to the air by the robot arm. Since the shape of the object is continuous, it is not possible to pre-collect all possible object poses for training of the classification algorithm. During the deformation of the object, when the key features are occluded, the existing object classification algorithm based on single-frame images cannot obtain high accuracy.

[0005] Existing object continuous deformation classification algorithms all use the number of single-frame classification results as the basis for judging the object classification result, or classify the object according to the single-frame picture, and then obtain the final classification result according to the maximum similarity.

[0006] The main disadvantage of the existing method is that the object in the single-frame picture may be in a folded state, making it difficult to extract features, so different object poses should have different contributions to the object classification result. If the object state stays in a certain specific pose for a long time, it is likely to cause an abnormal increase in the number of pictures of the wrong class, resulting in an incorrect object classification result.

[0007] Patent document CN115170594 A discloses a deformable object state representation method, including the following steps: acquiring an RGB image of a deformable object; obtaining the contour of the deformable object from the RGB image; obtaining the centroid of the contour using the obtained contour; establishing a polar coordinate system with the centroid as the center point and dividing a plurality of sector regions; calculating the average distance of the contour points in each sector region to the centroid as a one-dimensional feature in the multi-dimensional features of the deformable object state. This technical solution can realize the representation of the state of the deformable object, but cannot realize the classification of the deformable object.

[0008] Therefore, a continuous perception object classification algorithm considering inter-frame change information is needed to achieve high-accuracy object category classification. SUMMARY

[0009] In view of the above, the purpose of the present application is to provide a deformable object classification method and device based on optical flow change and continuous perception setting to achieve improved object classification robustness and accuracy while reducing the required data amount.

[0010] To achieve the above-mentioned purpose of the application, the present application provides a deformable object classification method based on optical flow change and continuous perception setting, comprising the following steps:

[0011] Obtaining a sequence of depth images and a sequence of video frames of a deformable object in a deformation process;

[0012] Using a depth convolution network to calculate the classification result of the deformable object in each frame of depth image;

[0013] Calculating the optical flow change between adjacent video frames according to the sequence of video frames;

[0014] Determining the effectiveness of the optical flow change between adjacent video frames according to the classification results of adjacent two frames of depth image;

[0015] Calculating the optical flow change cumulative value of each object category according to the effectiveness of the optical flow change;

[0016] Screening the object category with the largest optical flow change cumulative value as the final classification result determined in the deformation process.

[0017] Preferably, using a depth convolution network to calculate the classification result of the deformable object in each frame of depth image comprises:

[0018] For the depth image V t collected at the t time, using a depth convolution network as an image classifier ImageClassifier to classify the depth image V t , denoted as:

[0019] C t = ImageClassifier(V t ), C t ∈{1,…,K}

[0020] Wherein, C t represents the classification result of the t frame of depth image V t , and K represents the total number of categories that the deformable object can be classified into.

[0021] Preferably, the calculation of the optical flow change between adjacent video frames according to the sequence of video frames comprises:

[0022] At the t-th moment, according to the video frames I t and I t-1 , the inter-frame dense optical flow change is calculated:

[0023]

[0024] wherein, O t is a WxHx2-dimensional matrix storing the orthogonal direction optical flow values of all pixel points on the video frame, W and H are the width and height of the video frame respectively, and OpticalFlow(·) represents the dense optical flow calculation mode;

[0025] According to O t , the sum of the vector optical flow values of all pixel points in the video frame is calculated as the optical flow change M t between the adjacent two frames:

[0026]

[0027]

[0028] wherein, x, y represent the pixel coordinates in the video frame, represent the horizontal and vertical vector optical flow values in the video frame, which are extracted from O t , and ‖·‖2 represents the L2 norm.

[0029] Preferably, the determination of the effectiveness of the optical flow change between the adjacent video frames according to the classification results of the adjacent two frames of depth images comprises:

[0030] When the classification results of the t-th frame and the t-1-th frame of depth images are represented as C t-1 and C t , the effectiveness of the inter-frame optical flow change of the t-th frame and the t-1-th frame is represented as:

[0031]

[0032] wherein, k represents a category of the deformable object, and otherwise represents other cases.

[0033] Preferably, the calculation of the optical flow change cumulative value of each object category according to the effectiveness of the optical flow change comprises:

[0034] At the t-th moment, the optical flow change cumulative value of each object category is calculated and represented as:

[0035]

[0036] wherein, ​​an effectiveness of an inter-frame optical flow change amount between the t-th frame and the t-1-th frame M t an optical flow change amount M between the t-th frame and the t-1-th frame t k represents a category of a deformable object, and K represents a total number of categories to which the deformable object can be classified.

[0037] Preferably, an object category with a maximum optical flow change amount cumulative value is screened as a final classification result determined in the deformation process, including:

[0038] At a final T-th moment, a final classification result k is determined by using the following formula * (T):

[0039]

[0040] wherein, represents an optical flow change amount cumulative value at the T-th moment, k represents a category of a deformable object, and K represents a total number of categories to which the deformable object can be classified, and argmax(·) represents screening a maximum value.

[0041] To achieve the above-mentioned object, the embodiment further provides a deformable object classification device based on an optical flow change amount and continuous perception setting, including: an acquisition module, a classification module, an optical flow change amount calculation module, an effectiveness calculation module, a cumulative value calculation module, and a screening classification module.

[0042] The acquisition module is configured to acquire a sequence of depth images and a sequence of video frames of a deformable object in a deformation process.

[0043] The classification module is configured to calculate a classification result of the deformable object in each frame of depth image by using a deep convolutional network.

[0044] The optical flow change amount calculation module is configured to calculate an optical flow change amount between adjacent video frames according to the sequence of video frames.

[0045] The effectiveness calculation module is configured to determine an effectiveness of the optical flow change amount between the adjacent video frames according to classification results of adjacent two frames of depth image.

[0046] The cumulative value calculation module is configured to calculate an optical flow change amount cumulative value of each object category according to the effectiveness of the optical flow change amount.

[0047] The screening classification module is configured to screen an object category with a maximum optical flow change amount cumulative value as a final classification result determined in the deformation process.

[0048] To achieve the above object, the embodiment further provides a computing device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above deformable object classification method based on the optical flow change and the continuous perception setting when executing the computer program.

[0049] To achieve the above object, the embodiment further provides a computer readable storage medium, which stores a computer program, wherein the computer program is executable on a processor to implement the above deformable object classification method based on the optical flow change and the continuous perception setting.

[0050] Compared with the prior art, the present application has at least the following beneficial effects:

[0051] The classification result of the deformable object in each frame of depth image is determined through the depth image, the inter-frame optical flow change is calculated through the video frame, and the effectiveness of the optical flow change is determined in combination with the classification result, on the basis of which, the optical flow change cumulative value of each object category is calculated based on the effectiveness, and the object category with the maximum optical flow change cumulative value is screened as the final classification result determined in the deformation process. Such classification method determines the final classification result in combination with the inter-frame change in the deformation process, improves the high object classification robustness and accuracy, and all the data relied on are the data collected by the sensor, without using additional artificial auxiliary information. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0053] Figure 1 is a flow chart of the deformable object classification method based on the optical flow change and the continuous perception setting provided by the embodiment;

[0054] Figure 2 is a flow chart of the deformable object classification method based on the optical flow change and the continuous perception setting provided by the embodiment;

[0055] Figure 3 is a structural schematic diagram of the deformable object classification device based on the optical flow change and the continuous perception setting provided by the embodiment. DETAILED DESCRIPTION

[0056] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the protection scope of the present application.

[0057] In order to improve the robustness and accuracy of deformable object classification on the basis of small amount of data, the embodiments provide a deformable object classification method and device based on optical flow change and continuous perception setting.

[0058] Figure 1 and Figure 2 are a flowchart and a flow block diagram of the deformable object classification method based on optical flow change and continuous perception setting provided by the embodiments. As shown in Figure 1 and Figure 2 , the deformable object classification method based on optical flow change and continuous perception setting provided by the embodiments includes the following steps:

[0059] Step 1, obtaining a sequence of depth images and a sequence of video frames of a deformable object in a deformation process.

[0060] The deformable object will deform during being grabbed by a mechanical arm, and at the same time, depth data and video frame data of the deformable object in the deformation process are collected to form a sequence of depth images and a sequence of video frames. The number of depth images is the same as the number of video frames, and the depth image and the video frame at the same position in the sequence correspond to the same sampling time. The depth image and the video frame at each time are classified and the optical flow change is calculated at the same time.

[0061] Step 2, calculating the classification result of the deformable object in each depth image by using a depth convolution network.

[0062] In the embodiments, the classification result of the deformable object in each depth image is calculated by using a depth convolution network, including:

[0063] For the depth image V t collected at the t-th time, a depth convolution network is used as an image classifier ImageClassifier to classify the depth image V t , which is expressed as:

[0064] C t =ImageClassifier(V t ),C t ∈{1,…,K}

[0065] Wherein, C t represents the classification result of the t-th depth image V t , and K represents the total number of classes that the deformable object can be classified into.

[0066] For example, as shown in Figure 2 When the deformable object is a piece of clothing, the total number of categories in which the clothing can be classified is 5, namely trousers, shirts, sweaters, towels, and T-shirts.

[0067] Step 3, calculating the optical flow change between adjacent video frames according to the video frame sequence.

[0068] In the embodiment, the optical flow change between adjacent video frames is calculated according to the video frame sequence, comprising:

[0069] At the t-th moment, the inter-frame dense optical flow change is calculated according to the video frames I t and I t-1 .

[0070]

[0071] wherein O t is a WxHx2 matrix storing the orthogonal direction optical flow values of all pixel points on the video frame, W and H are the width and height of the video frame respectively, and OpticalFlow(·) represents the dense optical flow calculation method, specifically including the Horn-Schunck algorithm (see Horn B K P, Schunck B G. Determining optical flow [J]. Artificial intelligence, 1981, 17(1-3): 185-203.), LiteFlowNet algorithm (see Hui T W, Tang X, Loy CC. Liteflownet: A lightweight convolutional neural network for optical flow estimation [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 8981-8989.), PWCNet algorithm (see Sun D, Yang X, Liu M Y, et al. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 8934-8943.) and the like.

[0072] According to O tThe sum of the vector optical flow values of all pixels in the video frame is calculated as the optical flow change M between the two adjacent frames t :

[0073]

[0074] wherein x, y represent the pixel coordinates in the video frame, represent the vector optical flow values in the horizontal and vertical directions in the video frame, and are extracted from O t , and ‖·‖2 represents the L2 norm.

[0075] As shown in Figure 2 , the classification result at each time is corresponded to the optical flow change, which is convenient for the calculation of the optical flow change accumulation value in the subsequent step.

[0076] Step 4, determining the effectiveness of the optical flow change between the adjacent video frames according to the classification results of the adjacent two frames of depth images.

[0077] In the embodiment, the effectiveness of the optical flow change between the adjacent video frames is determined according to the classification results of the adjacent two frames of depth images, including:

[0078] When the classification results of the t-th frame and the t-1-th frame of depth images are represented as C t-1 and C t , the effectiveness of the inter-frame optical flow change of the t-th frame and the t-1-th frame is represented as:

[0079]

[0080] wherein k represents a category of the deformable object, and otherwise represents other cases.

[0081] Step 5, calculating the optical flow change accumulation value of each object category according to the effectiveness of the optical flow change.

[0082] In the embodiment, the optical flow change accumulation value of each category is calculated as the data is acquired frame by frame, and the optical flow change accumulation value of each object category is calculated according to the effectiveness of the optical flow change, including:

[0083] At the t-th time, the optical flow change accumulation value of each object category is represented as:

[0084]

[0085] As shown in Figure 2 ​​As shown, according to the effectiveness calculation of the optical flow change amount, the optical flow change amount cumulative value of the pants category, the shirt category and the towel category accounts for 0, the optical flow change amount cumulative value of the towel category accounts for 75, and the optical flow change amount cumulative value of the T-shirt category accounts for 25.

[0086] Step 6: screening the object category with the largest optical flow change amount cumulative value as the final classification result in the deformation process.

[0087] In the embodiment, steps 1 to 5 are repeated until the video frame and the depth image are terminated, that is, the effective optical flow change value is accumulated frame by frame until the video frame and the depth image are terminated, and then the object category with the largest optical flow change amount cumulative value is screened as the final classification result in the deformation process, including:

[0088] At the final T time, the final classification result k is determined by the following formula * (T):

[0089]

[0090] Wherein, The optical flow change amount cumulative value at the T time is represented, and argmax(·) represents screening the maximum value.

[0091] For example, as shown in Figure 2 , the sweater is screened as the final classification result.

[0092] The deformable object classification method provided by the embodiment takes the effective optical flow cumulative value as the evaluation index of the video classification result, which can avoid the interference of a large number of repeated frames in the deformable object grabbing process on the video classification result, and effectively improves the deformable object classification accuracy.

[0093] Based on the same inventive concept, the embodiment further provides a deformable object classification device based on optical flow change amount and continuous perception setting, as shown in Figure 3 , which comprises an acquisition module, a classification module, an optical flow change amount calculation module, an effectiveness calculation module, a cumulative value calculation module and a screening classification module.

[0094] The obtaining module is configured to obtain a sequence of depth images and a sequence of video frames of the deformable object in a deformation process; the classification module is configured to calculate a classification result of the deformable object in each depth image by using a deep convolutional network; the optical flow change amount calculation module is configured to calculate an optical flow change amount between adjacent video frames according to the sequence of video frames; the effectiveness calculation module is configured to determine the effectiveness of the optical flow change amount between the adjacent video frames according to the classification results of the adjacent two depth images; the cumulative value calculation module is configured to calculate an optical flow change amount cumulative value of each object category according to the effectiveness of the optical flow change amount; and the screening classification module is configured to screen an object category with the largest optical flow change amount cumulative value as a final classification result determined in the deformation process.

[0095] It should be noted that the deformable object classification device based on the optical flow change amount and the continuous perception setting provided in the above embodiments should be illustrated by the division of the above functional modules when classifying the deformable object, and the above functions can be completed by different functional modules according to the needs, that is, the internal structure of the terminal or the server is divided into different functional modules to complete all or part of the functions described above. In addition, the deformable object classification device based on the optical flow change amount and the continuous perception setting provided in the above embodiments and the deformable object classification method based on the optical flow change amount and the continuous perception setting belong to the same concept, and the specific implementation process is detailed in the deformable object classification method based on the optical flow change amount and the continuous perception setting. Embodiments, which will not be repeated here.

[0096] The embodiments also provide a computing device including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the deformable object classification method based on the optical flow change amount and the continuous perception setting when executing the computer program, including the following steps:

[0097] Step 1, obtaining a sequence of depth images and a sequence of video frames of the deformable object in a deformation process;

[0098] Step 2, calculating a classification result of the deformable object in each depth image by using a deep convolutional network;

[0099] Step 3, calculating an optical flow change amount between adjacent video frames according to the sequence of video frames;

[0100] Step 4, determining the effectiveness of the optical flow change amount between the adjacent video frames according to the classification results of the adjacent two depth images;

[0101] Step 5, calculating an optical flow change amount cumulative value of each object category according to the effectiveness of the optical flow change amount;

[0102] Step 6, screening an object category with the largest optical flow change amount cumulative value as a final classification result determined in the deformation process.

[0103] The memory can be a volatile memory, such as a RAM, at a near end, and can also be a non-volatile memory, such as a ROM, a FLASH, a floppy disk, a mechanical hard disk, and the like, and can also be a storage cloud at a far end. The processor can be a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), i.e., the steps of the deformable object classification method can be implemented through these processors.

[0104] The embodiment also provides a computer readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the deformable object classification method based on the optical flow change amount and the continuous perception setting, including the following steps:

[0105] Step 1, obtaining a sequence of depth images and a sequence of video frames of a deformable object in a deformation process;

[0106] Step 2, calculating a classification result of the deformable object in each depth image by using a depth convolutional network;

[0107] Step 3, calculating an optical flow change amount between adjacent video frames according to the sequence of video frames;

[0108] Step 4, determining the effectiveness of the optical flow change amount between the adjacent video frames according to the classification results of the adjacent depth images;

[0109] Step 5, calculating an optical flow change amount cumulative value of each object class according to the effectiveness of the optical flow change amount;

[0110] Step 6, screening an object class with the largest optical flow change amount cumulative value as a final classification result in the deformation process.

[0111] The computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, and the like.

[0112] The above detailed description of the specific embodiments has described the technical solutions and beneficial effects of the present application. It should be understood that the above description is only the most preferred embodiment of the present application, and is not used to limit the present application. Any modification, supplement, and equivalent replacement, and the like, within the principle range of the present application, should be included in the protection range of the present application.

Claims

1. A deformable object classification method based on an amount of optical flow change and a continuous perception setting, characterized by, The method comprises the following steps: obtaining a sequence of depth images and a sequence of video frames of a deformable object in a deformation process; calculating a classification result of the deformable object in each depth image by using a deep convolutional network; calculating an optical flow change between adjacent video frames according to the sequence of video frames, comprising: At the t-th moment, according to the video frame I of the t-th frame and the t-1-th frame t and I t-1 , the inter-frame dense optical flow change is calculated: where O t is a WxHx2 matrix storing the optical flow values of all pixels on the video frame in the orthogonal directions, W and H are the width and height of the video frame, respectively, and OpticalFlow(·) denotes the dense optical flow computation method. According to O t The sum of the vector optical flow values of all pixel points in the video frame is calculated as the optical flow change M between the two adjacent frames t : where x, y represent the pixel coordinates in the video frame, denote the horizontal and vertical vector optical flow values in the video frame, respectively, from O t are extracted, and || · ||2denotes the L2 norm. determining the validity of the optical flow change between the adjacent video frames according to the classification results of the adjacent two depth images, comprising: When the classification result of the t-th frame and the t-1-th frame depth images is represented as C t-1 and C t , the effectiveness of the inter-frame optical flow change amount of the t-th frame and the t-1-th frame is represented as: ​ wherein k represents a category of the deformable object, and otherwise represents other cases; calculating an optical flow change cumulative value of each object category according to the validity of the optical flow change; selecting an object category with the largest optical flow change cumulative value as the final classification result determined in the deformation process. 2.The deformable object classification method based on the optical flow change amount and the continuous perception setting of claim 1, wherein, calculating a classification result of the deformable object in each depth image by using a deep convolutional network, comprising: For the depth image V collected at the t time t , a depth convolution network is used as an image classifier ImageClassifier to classify the depth image V t , which is expressed as: C t = ImageClassifier(V t ), C t ∈ {1,..., K} where C t represents the classification result of the t-th frame depth image V t , and K represents the total number of classes in which the deformable object can be classified. 3.The deformable object classification method based on the optical flow change amount and the continuous perception setting of claim 1, wherein, calculating an optical flow change cumulative value of each object category according to the validity of the optical flow change, comprising: At the t-th time, the light flow change amount cumulative value of each object class is calculated is expressed as: wherein, represents the effectiveness of the inter-frame optical flow change amount between the t-th frame and the t-1-th frame M t represents the optical flow change amount M between the t-th frame and the t-1-th frame t , k represents a category of a deformable object, and K represents the total number of categories into which the deformable object can be classified. 4.The deformable object classification method based on the optical flow change amount and the continuous perception setting of claim 1, wherein, selecting an object category with the largest optical flow change cumulative value as the final classification result determined in the deformation process, comprising: At the final time T, the final classification result k is determined using the following formula * (T): wherein, denotes the optical flow change amount cumulative value at the T-th moment, k denotes a category of a deformable object, K denotes the total number of categories into which the deformable object can be classified, and argmax(·) denotes a maximum value screening.

5. A deformable object classification device based on an amount of optical flow change and a continuous perception setting, characterized by, comprising: an obtaining module, a classification module, an optical flow change calculation module, a validity calculation module, a cumulative value calculation module, and a screening classification module; the obtaining module is configured to obtain a sequence of depth images and a sequence of video frames of a deformable object in a deformation process; the classification module is configured to calculate a classification result of the deformable object in each depth image by using a deep convolutional network; the optical flow change calculation module is configured to calculate an optical flow change between adjacent video frames according to the sequence of video frames, comprising: At the t-th moment, according to the video frame I of the t-th frame and the t-1-th frame t and I t-1 , the inter-frame dense optical flow change is calculated: where O t is a WxHx2 matrix storing the optical flow values of all pixels on the video frame in the orthogonal directions, W and H are the width and height of the video frame, respectively, and OpticalFlow(·) denotes the dense optical flow computation method. According to O t The sum of the vector optical flow values of all pixel points in the video frame is calculated as the optical flow change M between the two adjacent frames t : where x, y represent the pixel coordinates in the video frame, denote the horizontal and vertical vector optical flow values in the video frame, from O t are extracted, and || · ||2denotes the L2 norm. the validity calculation module is configured to determine the validity of the optical flow change between the adjacent video frames according to the classification results of the adjacent two depth images, comprising: The classification result of the t-th frame and the t-1-th frame depth images is represented as C t-1 and C t The effectiveness of the inter-frame optical flow change amount of the t-th frame and the t-1-th frame is represented as: ​ wherein k represents a category of the deformable object, and otherwise represents other cases; the cumulative value calculation module is configured to calculate an optical flow change cumulative value of each object category according to the validity of the optical flow change; the screening classification module is configured to select an object category with the largest optical flow change cumulative value as the final classification result determined in the deformation process.

6. A computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, The processor executes the computer program to realize the deformable object classification method based on the optical flow change and the continuous perception setting according to any one of claims 1-4.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the deformable object classification method based on the optical flow change and the continuous perception setting according to any one of claims 1-4.

Images