Grasping robot and grasping method and system based on target and scene text matching

Abstract

The present invention belongs to the field of intelligent robots, and provides a grasping robot and a grasping method and system based on target and scene text matching. According to the image of the target to be grasped acquired by a camera and the target detection model, CNN is used to extract features, and regression is performed to obtain the classification result and boundary box of the target to be grasped; for targets with the same classification results, a text detection and recognition model is used to extract the text in the target detection box area for detection and recognition, and the initial three-dimensional coordinates are obtained when the text recognition result successfully matches the specific target; the target tracking algorithm is used to locate the specific grasping target detection box to obtain the final grasping coordinates, and the chassis movement and the robot arm action are controlled according to the grasping coordinates to grasp the specific target.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent robots, and in particular relates to a grasping robot based on target and scene text matching and a grasping method and system thereof. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In the prior art, most robot grasping detection algorithms directly perform grasping detection on a single object, or use complex neural networks to segment, classify, label, and other methods to distinguish multiple objects. However, when there are a large number of objects in the grasping scene, and the objects have consistent information such as appearance and color, or belong to the same category of objects, the above detection algorithms cannot distinguish such objects in detail, which directly affects the robot's grasping judgment and leads to insufficient grasping accuracy. Summary of the invention

[0004] In order to solve at least one technical problem existing in the above-mentioned background technology, the present invention provides a grasping robot and a grasping method and system based on target and scene text matching, which integrates the target detection model and the text detection and recognition model to achieve matching of the text recognition result with the object target detection frame, completes the precise positioning of the object to be grasped, and realizes real-time tracking of specific targets through the KCF tracking algorithm, thereby realizing precise grasping control of the robot.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A first aspect of the present invention provides a grasping robot based on target and scene text matching, comprising: a depth camera, a chassis, a mechanical arm, and a controller;

[0007] The controller includes a preliminary detection module for the target to be captured, a text detection and recognition module, and a target capture module;

[0008] The preliminary detection module for the target to be captured is configured to: extract features using CNN based on the target image to be captured and the target detection model acquired by the camera, and regress to obtain the classification result and bounding box of the target to be captured;

[0009] The text detection and recognition module is configured to: for targets with the same classification results, use the text detection and recognition model to extract the text in the target detection box area for detection and recognition, and obtain the initial three-dimensional coordinates when the text recognition result successfully matches the specific target;

[0010] The target grabbing module is configured to: use a target tracking algorithm to locate a specific grabbing target detection frame, obtain final grabbing coordinates, and control chassis movement and robotic arm action to grab the specific target according to the grabbing coordinates.

[0011] A second aspect of the present invention provides a crawling method based on target and scene text matching, comprising the following steps:

[0012] Obtain the target image to be captured;

[0013] According to the target image and target detection model, CNN is used to extract features and regress the classification results and bounding boxes of the target.

[0014] For targets with the same classification results, the text detection and recognition model is used to extract the text in the target detection box area for detection and recognition. When the text recognition result successfully matches the specific target, the initial three-dimensional coordinates are obtained;

[0015] The target tracking algorithm is used to locate the detection frame of a specific grasping target and obtain the final grasping coordinates. The chassis movement and the robot arm action are controlled according to the grasping coordinates to grasp the specific target.

[0016] A third aspect of the present invention provides a crawling system based on target and scene text matching, comprising:

[0017] The robot includes a preliminary detection module for the target to be grasped, a text detection and recognition module, and a target grasping module;

[0018] The preliminary detection module for the target to be captured is used to obtain the image of the target to be captured; based on the image of the target to be captured and the target detection model, CNN is used to extract features and regress to obtain the classification result and bounding box of the target to be captured;

[0019] The text detection and recognition module is used to extract the text in the target detection box area for detection and recognition using the text detection and recognition model for targets with the same classification results, and obtain the initial three-dimensional coordinates when the text recognition result successfully matches the specific target;

[0020] The target grabbing module is used to use the target tracking algorithm to locate the specific grabbing target detection frame, obtain the final grabbing coordinates, and control the chassis movement and the robot arm movement to grab the specific target according to the grabbing coordinates.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The present invention uses the lightweight target detection model NanoDet to perform target detection on the object to be grasped, and then enhances the image in the detection frame area to overcome unfavorable factors such as the target area being too small. The text detection and recognition model PP-OCR is used to perform text detection and recognition on the enhanced detection frame area to extract text information. The target information provided by the two models is fused to match the text recognition results with the object target detection frame to complete the precise positioning of the object to be grasped. The real-time tracking of specific targets is achieved through the KCF tracking algorithm, thereby realizing the precise grasping control of the robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0024] Figure 1 This is a schematic diagram of the overall process of the target capture, monitoring and positioning method according to the first embodiment of the present invention;

[0025] Figure 2 This is a structural diagram of NanoDet according to Embodiment 1 of the present invention;

[0026] Figure 3 1 is a schematic diagram of an image enhancement process in a detection frame area according to a first embodiment of the present invention;

[0027] Figure 4 This is a schematic diagram of the PP-OCR detection process in Example 1 of the present invention;

[0028] Figure 5 This is a CRNN structure diagram of the first embodiment of the present invention;

[0029] Figure 6(a)-Figure 6(b) This is the IOU calculation process of the first embodiment of the present invention;

[0030] Figure 7(a)-Figure 7(d) This is the target tracking process of the first embodiment of the present invention;

[0031] Figure 8(a)-Figure 8(c) This is the depth camera calibration and registration process of the first embodiment of the present invention;

[0032] Figure 9(a)-Figure 9(c) This is the grabbing action of the robot arm in the first embodiment of the present invention;

[0033] Figure 10(a)-Figure 10(e) This is a diagram showing the effect of text detection and recognition in Example 1 of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0035] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0036] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0037] Embodiment 1

[0038] The present invention proposes a method of utilizing the text information of the grasped object, integrating the relevant information provided by the target detection and text detection and recognition algorithms, and constructing an accurate detection system for specific grasped targets, thereby realizing accurate recognition and positioning of the target objects. At the same time, a lightweight model is adopted to ensure the real-time effect of the system, which is convenient for the deployment of grasping tasks on the robot controller, and solves the problem that the current grasping target detection algorithm cannot distinguish similar objects in detail.

[0039] like Figure 1 As shown, this embodiment provides a grasping robot based on target and scene text matching, including a depth camera, a chassis, a mechanical arm and a controller;

[0040] The controller includes a preliminary detection module for objects to be captured and a text detection and recognition module;

[0041] The depth camera is used to capture the image of the target to be captured, and the preliminary detection module of the target to be captured is configured to: perform feature extraction using CNN according to the image of the target to be captured and the target detection model, and regress to obtain the classification result category and bounding box of the target to be captured.

[0042] The text detection and recognition module is configured as follows: for targets with the same classification results, a text detection and recognition model is used to extract text in the target detection frame area for detection and recognition; when the text recognition result successfully matches the specific target, the initial three-dimensional coordinates are obtained; the target tracking algorithm is used to locate the specific grasping target detection frame to obtain real-time grasping coordinates; and the chassis movement and the robotic arm action are controlled according to the grasping coordinates to grasp the specific target.

[0043] In this embodiment, the target detection model adopts NanoDet, which is a high-speed, lightweight anchor-free target detection model that can provide performance close to that of the YOLO series and is also easy to train and transplant.

[0044] The network structure of the target detection model is as follows Figure 2 As shown in the figure. NanoDet is a FCOS (Fully Convolutional One-Stage Object Detection) style detection network. The model can be divided into three parts: backbone network, feature fusion layer and detection head. In order to ensure that the model parameter volume is as small as possible, the backbone network uses ShuffleNetV2.0, and the last layer of convolution of ShuffleNetV2.0 is removed. The features of 8, 16, and 32 times downsampling are extracted and input into PAN for multi-scale feature fusion.

[0045] The feature fusion layer uses PAN, which adds a bottom-up approach to downsample low-order feature maps and then adds the downsampled results to high-order features. The detection head of NanoDet uses two 96-channel convolutional layers, using the same set of convolution calculations for bounding box regression and classification.

[0046] The target detection module can obtain the detection frame of the object in real time and classify the image targets within the detection frame, thereby locating the two-dimensional position of each object in the image. For objects with the same classification results, the detection system will further distinguish them by extracting the text information on the object.

[0047] Since the camera captures the entire scene image, the area where the text is located is small and is often affected by other factors such as lighting. If the entire image is input into the text detection network, the features of the text area cannot be fully extracted, resulting in poor text area detection results.

[0048] In order to improve the accuracy of subsequent text detection and recognition, the text area image enhancement operation is performed before text detection.

[0049] like Figure 3 As shown, the controller also includes a text area image enhancement module, which is configured to: image cropping, image magnification and filling, grayscale processing and image sharpening.

[0050] (1) Image cropping: Cropping each target object from the entire image based on the target bounding box generated by target detection;

[0051] (2) Image enlargement and filling: Since each cropped target area is too small, this paper uses the bicubic interpolation method to enlarge the cropped area to twice its original size and fill the border of the enlarged image to fill each detection box area with a square image with the same aspect ratio;

[0052] (3) Grayscale processing: The enlarged and filled image is grayscaled to remove the influence of variables such as color and illumination. Then the grayscale image is transformed using histogram equalization to increase the contrast of the text area.

[0053] (4) Image sharpening: Finally, the image sharpening method is used to enhance the edge of the text in the image to make the text clearer.

[0054] After the above processing, each target detection box area has the same shape and is enhanced, which is used as the input of the text detection and recognition model.

[0055] like Figure 4 As shown, the text detection and recognition model includes a text detection module, a detection frame correction module and a text recognition module.

[0056] The text detection module is configured as follows:

[0057] Locate the text area in the image and use DB-Net as the text detector. DB-Net will binarize the standard binarization in the segmentation network. i,j Change to differentiable binarization function

[0058]

[0059]

[0060] In both formulas, B i,j , is a binary image, P i,j is the probability graph, t, T i,j To set the threshold.

[0061] Differentiable binarization solves the problem of non-differentiable gradients in standard binarization during training. To further improve efficiency, PP-OCR uses six strategies to slim down DB-Net.

[0062] The detection frame correction module is configured as follows:

[0063] Before recognizing the text in the detection box, the detection box needs to be corrected. A text direction classifier is designed in PP-OCR. First, the text detection box is converted into a horizontal rectangular box through geometric transformation, and then the direction of the converted text is determined. If the text box is reversed, it is further flipped. At the same time, four strategies are used to enhance the model capability and reduce the model size.

[0064] The text recognition module is configured as follows: using CRNN as a text recognizer, the CRNN network structure is as follows Figure 5As shown in the figure, CRNN combines feature extraction and sequence modeling, and uses CTC (Connectionist Temporal Classification, CTC) loss for sequence alignment. In order to enhance the ability of text recognition and reduce the size of the model, nine strategies are used in this paper to process it.

[0065] In this embodiment, the text detection and recognition model uses an ultra-lightweight PP-OCR text detection and recognition network, making it easier to deploy on mobile terminals. Through PP-OCR text detection and recognition, the text information contained in all target objects in the target detection is obtained. Even if the objects are classified as the same type by the target detection module, they can be further distinguished according to the recognized text information.

[0066] The configuration for acquiring the initial three-dimensional coordinates is as follows: by determining whether the given text information contained in the recognition result text belongs to the corresponding actual object in the detection box, if so, the text is matched with the target object, and the initial three-dimensional coordinates of the captured target are calculated based on the matched target boundary box coordinates combined with the camera depth information.

[0067] The above-mentioned target preliminary detection module and text detection and recognition module are performed in units of image frames. Assume that the current frame image F i After the above process, multiple regions are extracted with target detection boxes as units, namely D 1 ,D 2 ,…,D n , the actual object in the detection frame is d 1 ,d 2 ,…,d n , a specific text information t to be matched is given in the task, and n detection box areas are subjected to text detection and recognition, and the recognition result is [T 1 ,T 2 ,…,T n ], if in a certain recognition result text T t If the given text information t is included, the following judgment can be made:

[0068]

[0069]

[0070] Through the above formula, it is determined that the text information t belongs to the object d t , that is, the matching of text and target object is completed, and the initial three-dimensional coordinates of the captured target can be obtained by combining the detected position of the target object at this moment with the depth information.

[0071] According to the physical characteristics of the mobile service robot, the robot uses the initial three-dimensional coordinates as the first coordinate input, moves the chassis, and prepares for the grasping action. The movement of the robot chassis will cause the image captured by the camera to change in real time. Then the detection frame located according to the text recognition result at the previous moment has been relatively offset, and the robot needs to receive the new coordinate position of the object in real time. If the text detection and recognition is performed on the new detection frame area in each frame, the network calculation amount will be huge, which will cause the real-time effect of the entire system to deteriorate, and directly affect the robot's grasping efficiency. Therefore, a tracking algorithm is used to track the target in real time.

[0072] In order to solve the problem of target position change caused by robot movement, the target tracking algorithm based on kernel correlation filter KCF (Kernel Correlation Filter, KCF) tracking algorithm is introduced and configured as follows:

[0073] By constructing a circulant matrix for the acquired image blocks to represent the samples of dense sampling of the target and its background, a large number of training sets are constructed. After the first frame image undergoes two-stage detection, the detection frame of the target object to be captured is located. At this time, the KCF algorithm is used to track the located target detection frame area in real time. The training of the tracker is to find the filter ω that minimizes the objective function;

[0074] The steps to solve ω are as follows:

[0075] (1) The ridge regression equation was constructed:

[0076]

[0077] X t is a single training sample extracted, y t is the corresponding credibility sample label, and λ is the regularization parameter to prevent overfitting of regression.

[0078] The cyclic shift of a single training sample constitutes the sample set X, which is the following cyclic matrix:

[0079]

[0080] (2) In the ridge regression equation, f(X) = ω T X, taking the derivative of the equation with respect to ω, we get:

[0081] ω=(X T X+λI) -1 X T Y

[0082] Among them, X T is the transpose of the training sample X, I is the identity matrix, Y is the column vector, and the label y t constitute.

[0083] The circulant matrix X has the property of being diagonalizable in Fourier space. Substitute the following Fourier diagonalization equation into the ridge regression:

[0084]

[0085] Where F is the discrete Fourier matrix, x represents the value of the first row of X after the Fourier transform, F H is the conjugate transposed matrix of F.

[0086] After a series of transformations, we can get:

[0087]

[0088] According to Fourier space transformation, we have:

[0089]

[0090] In the formula, F -1 is the inverse Fourier transform.

[0091] Using the target tracking algorithm, the detection frame positioning of a specific captured target includes:

[0092] When the robot is performing a grasping task and moving, considering that the KCF tracking algorithm may cause drift when errors accumulate over a long period of time, this embodiment calculates the intersection over union (IOU) between the tracking frame in each frame image and all current target detection frames, and searches for the target detection frame that maximizes the calculation result, so that the bounding box of the target to be grasped can be located in each frame image.

[0093] The IOU calculation diagram is as follows Figure 6(a)-Figure 6(b) As shown, the IOU calculation formula is:

[0094]

[0095] Assume that at this time, the target detection generates n detection boxes with the same classification label results, namely A 1 ,A 2 ,……,A n .

[0096] Through text detection and recognition, locate A t The object in the detection frame contains specific text information, that is, A t The target object in the detection frame is the target to be grasped. At this time, the KCF tracking algorithm is used to track A t The target in the image is sampled and the tracking frame T is generated. During the movement of the robot, the target is tracked in real time. During the whole process, T and A are calculated. i(i=1,2,…,n) IOU, the value that maximizes the IOU value at each moment is A t , the calculation formula is as follows:

[0097]

[0098] like Figure 7(a)-Figure 7(d) It represents the complete process of target detection frame tracking from time T0 to time T3.

[0099] By finding the detection frame with the largest IOU with the tracking frame, the real-time positioning of the grasped target detection frame can be achieved, so that the real-time position of the grasped target can be updated, and the robot completes the grasping task according to the grasping position. The positioning strategy of the robot grasping the target only needs to perform text detection and recognition once to achieve real-time positioning of the detection frame, which reduces the overall calculation amount and ensures real-time performance.

[0100] like Figure 8(a)-Figure 8(c) As shown, the depth camera calibration and registration process is configured as follows:

[0101] Using an 8×11 chessboard, the depth camera is calibrated using the Zhang Zhengyou calibration method for RGB and depth images, and the internal parameter matrices of the RGB camera and the depth camera are obtained as H rgb With H ir The external parameter matrix consists of a rotation matrix and a translation vector, which are R rgb 、T rgb With R ir 、T ir .

[0102] Let P rgb With P ir These are the spatial coordinates of a point under the RGB camera and depth camera coordinates. Since the coordinates of the depth camera are different from those of the RGB camera, the left relationship between the two can be linked by a rotation matrix and a translation vector:

[0103] P rgb =RP ir +T

[0104] By calculation and derivation, the rotation matrix R and translation vector T can be expressed as:

[0105]

[0106]

[0107] The camera coordinates are transformed using the calculated rotation matrix and translation vector to align the RGB-D images. According to the actual alignment situation, the translation vector between the two cameras is manually fine-tuned to obtain a better alignment effect.

[0108] like Figure 9(a)-Figure 9(c) As shown, it is a robot arm grasping action. After positioning the specific grasping target detection frame, it is configured as follows:

[0109] The two-dimensional coordinates of the center area of ​​the grasping detection frame and the depth information of the corresponding area of ​​the registered depth map are used as the original three-dimensional coordinate information of the grasped object, and the transformation matrix between the camera coordinate system and the robotic arm coordinate system is calculated. The three-dimensional coordinates obtained by the camera are mapped to the robotic arm coordinate system, which is the grasping coordinates of the robot. The robot moves to the reachable range of the robotic arm according to the grasping coordinates received in real time, and the robotic arm performs the grasping action, and the robot completes the grasping task.

[0110] The present invention integrates two detection algorithms, target detection and text detection and recognition, and integrates text information on the basis of the location information provided by the target detection algorithm, thereby realizing accurate detection of specific target objects. The present invention uses a lightweight deep learning model to build a detection system, which is easy to deploy on the robot controller and achieves real-time effects on the robot side. For the patient medicine bottle grabbing scene in the hospital, it is proved through design experiments that the method proposed by the present invention has high feasibility. The robot completes real-time detection and positioning of specific targets by identifying specific text information on the medicine bottle, realizing the robot's intelligent grabbing task of medicine bottles in the hospital scene.

[0111] Embodiment 2

[0112] This embodiment provides a crawling method based on target and scene text matching, comprising the following steps:

[0113] Step 1: Get the target image to be captured;

[0114] Step 2: Based on the target image and target detection model, CNN is used to extract features and regress the classification results and bounding boxes of the target.

[0115] Step 3: For targets with the same classification results, the text detection and recognition model is used to extract the text in the target detection box area for detection and recognition. When the text recognition result successfully matches the specific target, the initial three-dimensional coordinates are obtained;

[0116] Step 4: Use the target tracking algorithm to locate the specific grasping target detection frame and obtain the final grasping coordinates. According to the grasping coordinates, the chassis movement and the robot arm action are controlled to grasp the specific target.

[0117] Embodiment 3

[0118] This embodiment provides a grasping system based on target and scene text matching, including: a robot and a terminal, wherein the robot is used to receive a grasping instruction issued by the terminal;

[0119] The robot includes a preliminary detection module for the target to be grasped, a text detection and recognition module, and a target grasping module;

[0120] The preliminary detection module for the target to be captured is used to obtain the image of the target to be captured; based on the image of the target to be captured and the target detection model, CNN is used to extract features and regress to obtain the classification result and bounding box of the target to be captured;

[0121] The text detection and recognition module is used to extract the text in the target detection box area for detection and recognition using the text detection and recognition model for targets with the same classification results, and obtain the initial three-dimensional coordinates when the text recognition result successfully matches the specific target;

[0122] The target grabbing module is used to use the target tracking algorithm to locate the specific grabbing target detection frame, obtain the final grabbing coordinates, and control the chassis movement and the robot arm movement to grab the specific target according to the grabbing coordinates.

[0123] Taking the service robot grasping scenario in a medical environment as an example, first, the robot is commanded to grasp the medicine bottle, that is, the specific patient name information is sent to the robot. The target detection module detects and frames all the medicine bottles in the robot's field of view, and obtains the bounding box position of each medicine bottle. After that, the image enhancement operation extracts the target image in the detection frame area and enhances it. The enhanced image is sent to the text detection and recognition module for text detection and recognition. Finally, according to the text recognition results, it is matched with the given patient name information. The text detection effect is as follows: Figure 10(a)-Figure 10(e) shown.

[0124] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A grasping robot based on target and scene text matching, characterized in that: include: Depth camera, chassis, robotic arm and controller; the controller includes a preliminary detection module for the target to be captured, a text detection and recognition module and a target capture module; The target preliminary detection module is configured to: extract features using CNN based on the target image and target detection model acquired by the depth camera, and regress to obtain the classification result and bounding box of the target; the target preliminary detection module obtains the detection box of the object in real time, and classifies the image targets in the detection box, and locates the two-dimensional position of each object in the image; The text detection and recognition module is configured to: for targets with the same classification results, use the text detection and recognition model to extract the text in the target detection box area for detection and recognition, and obtain the initial three-dimensional coordinates when the text recognition result successfully matches the specific target; The target grabbing module is configured to: use the target tracking algorithm to locate the specific grabbing target detection frame, obtain the final grabbing coordinates, and control the chassis movement and the robot arm movement to grab the specific target according to the grabbing coordinates; after locating the specific grabbing target detection frame, it is configured to: use the located two-dimensional coordinates of the center area of ​​the grabbing detection frame and the depth information of the corresponding area of ​​the aligned depth map as the original three-dimensional coordinate information of the grabbing object, calculate the transformation matrix between the camera coordinate system and the robot arm coordinate system, and map the three-dimensional coordinates obtained by the camera to the robot arm coordinate system, which is the final grabbing coordinates; The target preliminary detection module and text detection and recognition module are performed in units of image frames. Suppose that the current frame image extracts multiple regions in units of target detection frames, namely D1, D2, ..., D n , the actual objects in the detection box are d1, d2, ..., d n , a specific text information t to be matched is given in the task, and n detection box areas are subjected to text detection and recognition, and the recognition results are [T1, T2, …, T n ], if in a certain recognition result text T t If the given text information t is included, the following judgment can be made: Through the above formula, it is determined that the text information t belongs to the object d t , that is, the text and the target object are matched, and the initial three-dimensional coordinates of the captured target can be obtained by combining the target object detection position and depth information at this moment; The initial three-dimensional coordinates are used as the first coordinate input, and the chassis is moved to prepare for the grasping action; When the robot is performing the grasping task, it calculates the intersection and union ratio of the tracking frame in each frame image with all current target detection frames to find the target detection frame that maximizes the calculation result, that is, the bounding box of the target to be grasped can be located in each frame image.

2. The grasping robot based on target and scene text matching as claimed in claim 1, characterized in that: The target tracking algorithm is configured as follows: introducing a target tracking algorithm based on a kernel correlation filter, a KCF tracking algorithm, constructing a circulant matrix for the acquired image blocks to characterize samples of dense sampling of the target and its background, constructing a large number of training sets, performing training, and finding a filter with the minimum objective function.

3. The grasping robot based on target and scene text matching as claimed in claim 1, characterized in that: The target detection model is configured as follows: using a NanoDet network, including a backbone network, a feature fusion layer and a detection head, the backbone network using ShuffleNetV2.0, and the feature fusion layer using PAN.

4. The grasping robot based on target and scene text matching as claimed in claim 1, characterized in that: The text detection and recognition model adopts the PP-OCR text detection and recognition network.

5. The grasping robot based on target and scene text matching as claimed in claim 1, characterized in that: The depth map is obtained by depth camera calibration and registration, and the depth camera calibration and registration process is configured as follows: The depth camera is calibrated using the Zhang Zhengyou calibration method to perform RGB and depth map calibration to obtain the intrinsic parameter matrix and extrinsic parameter matrix of the RGB camera and the depth camera, wherein the extrinsic parameter matrix is ​​composed of a rotation matrix and a translation vector; The camera coordinates are transformed according to the obtained rotation matrix and translation vector to obtain a depth map.

6. The grasping robot based on target and scene text matching as claimed in claim 1, characterized in that: The controller further comprises a text area image enhancement module, and the text area image enhancement module is configured to perform image cropping, image enlargement and filling, grayscale processing and image sharpening on the target image to be captured.

7. The grasping robot based on target and scene text matching as claimed in claim 1, characterized in that: The text recognition result and the specific target matching include: according to the given specific text information to be matched, multiple areas in the target detection frame are compared with the corresponding actual objects in the detection frame, and it is determined whether the given text information contained in the recognition result text belongs to the corresponding actual object object in the detection frame. If so, the matching of the text and the target object is completed.

8. A crawling method based on matching target and scene text, characterized in that: The method is applied to the grasping robot based on target and scene text matching as claimed in claim 1, comprising the following steps: Obtain the target image to be captured; According to the target image and target detection model, CNN is used to extract features and regress the classification results and bounding boxes of the target. For targets with the same classification results, the text detection and recognition model is used to extract the text in the target detection box area for detection and recognition. When the text recognition result successfully matches the specific target, the initial three-dimensional coordinates are obtained; The target tracking algorithm is used to locate the detection frame of a specific grasping target and obtain the final grasping coordinates. The chassis movement and the robot arm action are controlled according to the grasping coordinates to grasp the specific target.

9. A crawling system based on matching of target and scene text, adopting the crawling method based on matching of target and scene text as claimed in claim 8, characterized in that: The system is applied to the grasping robot based on target and scene text matching as claimed in claim 1, comprising: a preliminary detection module for the target to be grasped, a text detection and recognition module, and a target grasping module; The preliminary detection module for the target to be captured is used to obtain the image of the target to be captured; based on the image of the target to be captured and the target detection model, CNN is used to extract features and regress to obtain the classification result and bounding box of the target to be captured; The text detection and recognition module is used to extract the text in the target detection box area for detection and recognition using the text detection and recognition model for targets with the same classification results, and obtain the initial three-dimensional coordinates when the text recognition result successfully matches the specific target; The target grabbing module is used to use the target tracking algorithm to locate the specific grabbing target detection frame, obtain the final grabbing coordinates, and control the chassis movement and the robot arm movement to grab the specific target according to the grabbing coordinates.

Images