Spacecraft assembly identification and positioning method based on target detection and composite target code

Abstract

The present invention relates to a method for identifying and locating spacecraft assemblies based on target detection and composite target codes, comprising: calibrating a camera using the Zhang Zhengyou camera calibration method to correct captured images; identifying and roughly locating target assemblies in the images using an improved YOLOv3 target detection algorithm; moving the camera so that its visual center is aligned with the center of the target code of the target assembly; identifying the target code, obtaining corresponding assembly information, and verifying the recognition result of the target detection algorithm; and if the recognition result is correct, using composite target code technology to finely locate the target assembly based on the target code. By introducing the target detection algorithm and composite target code technology, this method achieves the identification and positioning of spacecraft assemblies, thereby improving the accuracy and stability of spacecraft assembly.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a method for identifying and positioning spacecraft assemblies based on target detection and composite target code technology. Background Art

[0002] With the advancement of Industry 4.0, spacecraft metrology and testing are moving towards intelligent and automated development. Since spacecraft parts are characterized by a wide variety and small batch sizes, their assembly process often involves long processing routes and mixed production lines for multiple models. Therefore, the development and application of automated assembly platforms are particularly important.

[0003] Automated assembly platforms have been applied in a variety of fields, including automotive assembly, special-shaped mold assembly, and automated light bulb assembly. Related research areas include virtual reality-based assembly and disassembly assistance systems, high-precision automated assembly systems based on vision and force information, dual-arm collaborative robot automated assembly systems, and flexible automated assembly systems suitable for small-batch, multi-variety assembly. Research related to spacecraft component assembly includes recognition and positioning of rivets by robotic grasping using binocular cameras, automated measurement of assembly accuracy in spacecraft assembly integration and experiments, positioning mechanism design in spacecraft assembly, and assembly systems for shaft-and-hole parts.

[0004] my country's aerospace industry primarily uses traditional methods to complete spacecraft assembly, resulting in relatively poor accuracy and stability, making it difficult to meet international standards. Automated assembly platforms could effectively address this issue, but their practical applications are limited, particularly the lack of an intelligent system that can be applied to a wide range of assembly tasks. Among the key technologies for automated assembly platforms, with the exception of system calibration and QR code encoding and decoding, which have mature specifications and applications, other technologies still have significant room for development and advancement, and their applicability in systems has yet to be verified.

[0005] During the assembly of spacecraft components, precise identification and positioning of the components is essential. Currently, the main precision positioning methods used domestically and internationally include laser, image processing, and QR code positioning. Laser methods can cause unpredictable damage to spacecraft components, making image processing and QR code positioning more suitable. However, while image processing can effectively identify spacecraft components, it struggles to obtain accurate positioning data. Existing QR code positioning methods rely on the fixed position of the spacecraft, making them cumbersome to operate. Summary of the Invention

[0006] In view of the above-mentioned deficiencies in existing spacecraft assembly, the purpose of the present invention is to provide a spacecraft assembly identification and positioning method based on target detection and composite target code, which improves the accuracy and stability of spacecraft assembly identification and positioning.

[0007] In order to achieve the above-mentioned object of the invention, the technical solution of the present invention is:

[0008] The present invention provides a method for identifying and locating spacecraft assemblies based on target detection and composite target codes, comprising:

[0009] Use Zhang Zhengyou's camera calibration method to calibrate the camera and correct the captured image;

[0010] Use the improved YOLOv3 target detection algorithm to identify and roughly locate the target assembly in the image;

[0011] Move the camera so that the visual center is aligned with the center of the target code of the target assembly;

[0012] Identify the target code, obtain corresponding assembly information, and verify the recognition result of the target detection algorithm;

[0013] If the recognition result is correct, the target assembly is precisely positioned according to the target code using composite target code technology.

[0014] According to one aspect of the present invention, the camera is calibrated using the Zhang Zhengyou camera calibration method to correct the captured image, including:

[0015] Print a flat calibration plate consisting of a checkerboard and fix it;

[0016] Placing the plane calibration plate at different positions and using a camera to capture images of different postures;

[0017] The features of the plane calibration plate are detected to obtain the internal and external parameters and distortion parameters of the camera, and to optimize and estimate the imaging accuracy.

[0018] According to one aspect of the present invention, an improved YOLOv3 target detection algorithm is used to identify and roughly locate a target assembly in an image, including:

[0019] Construct a dataset of assembly image data, which is divided into training set and test set;

[0020] Build an improved YOLOv3 target detection network;

[0021] Use the training set to train the improved YOLOv3 target detection network, iterate until the loss function no longer decreases, and obtain the weight file;

[0022] The weight file is used to test the images in the test set, identify the target assembly, and roughly estimate the pose of the target assembly in the standard coordinate system.

[0023] According to one aspect of the present invention, constructing a dataset of assembly part images includes:

[0024] Collect assembly images containing different workpiece types and different shooting distances;

[0025] Performing data format conversion and proportional scaling preprocessing on the assembly part images to form a basic data set;

[0026] Expanding the image data of the images in the basic data set by performing an image rotation operation, and performing data enhancement in combination with a proximity interpolation method to obtain an enhanced data set;

[0027] The images in the enhanced data set are marked, and the starting point coordinates, length, width and classification information of the images are recorded to obtain a marked enhanced data set.

[0028] According to one aspect of the present invention, an improved YOLOv3 target detection network is constructed, including: using GIOU instead of interaction ratio as an indicator for measuring target detection effect, and using Mish activation function instead of ReLU activation function.

[0029] According to one aspect of the present invention, moving the camera so that the visual center is aligned with the center of the target code of the target assembly comprises:

[0030] Use visual sensors to collect visual information around the robotic arm and determine its position and posture information;

[0031] Use force protection sensors to generate feedback information when the robot arm collides;

[0032] Perform obstacle detection, target recognition, and posture analysis based on posture information and feedback information, and make decisions;

[0033] The robot arm is controlled to move the camera to a specified position according to the decision information, so that the visual center is aligned with the center of the target code of the target assembly.

[0034] According to one aspect of the present invention, identifying the target code, obtaining corresponding assembly part information, and verifying the target detection recognition result include:

[0035] Using composite target code technology to identify information in the target code image captured by the camera;

[0036] According to the index information of the information storage layer of the target code, query the database for corresponding assembly information;

[0037] The assembly information is compared with the recognition result of the target detection algorithm. If the recognition result is wrong, the target assembly is re-detected.

[0038] According to one aspect of the present invention, the target assembly is precisely positioned according to the target code using composite target code technology, including:

[0039] Calculating the distance between the camera and the center point of the target code, and establishing a target code coordinate system based on the target code image captured by the camera;

[0040] Extracting corner points in the target code image using Harris corner detection algorithm;

[0041] Calculating the rotation angle of the target code image in space according to the corner points;

[0042] Determining the precise positional relationship between the assembly information and the target assembly based on the assembly information corresponding to the target code, and determining the precise positions of the corner point and the rotation angle in the target code coordinate system;

[0043] The precise position of the corner point in the standard coordinate system is calculated according to the rotation angle.

[0044] According to one aspect of the present invention, extracting corner points in the target code image using a Harris corner detection algorithm includes:

[0045] Calculate the horizontal and vertical gradients of the pixels of the target code image and their product to generate a matrix:

[0046] Performing Gaussian filtering on the target code image to obtain a new matrix;

[0047] Calculate the interest value of each pixel corresponding to the original target code image;

[0048] Select the pixel point corresponding to the local maximum interest value;

[0049] Set the threshold and select the corner point (x i,j ,y i,j ,0), where i is the i-th corner point in the horizontal direction and j is the j-th corner point in the vertical direction.

[0050] According to one aspect of the present invention, the precise position of the corner point in the standard coordinate system is calculated according to the rotation angle, and the specific calculation formula is as follows:

[0051]

[0052] Among them, the coordinates of the exact position are (x i ' ,j ,y i ' ,j ,z i ' ,j ); the coordinates of the corner points are (x i,j ,y i,j,0), i is the i-th corner point in the horizontal direction, j is the j-th corner point in the vertical direction; α represents the rotation angle of the longitudinal adjacent corner points of the target code image along the x-axis direction, β represents the rotation angle of the transverse adjacent corner points of the target code image along the y-axis direction, and θ represents the average value of the rotation angle of the adjacent corner points in all rows and columns of the target code image along the z-axis direction.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] The present invention addresses shortcomings in existing spacecraft assembly processes by introducing a target detection algorithm and a novel composite target code to achieve identification and location of spacecraft components, improving the accuracy and stability of spacecraft assembly. This technology utilizes an improved YOLOv3 target detection algorithm, which offers rapid detection speed and excellent accuracy, for identification and coarse location of components. Furthermore, the novel composite target code technology, combined with the target detection algorithm, enables identification and precise location of components. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be derived from these drawings without inventive effort.

[0056] Figure 1 A flowchart schematically illustrates an implementation of a method for identifying and locating spacecraft assemblies based on target detection and composite target codes, provided by an embodiment of the present invention;

[0057] Figure 2 Schematically showing a schematic diagram of a composite target code provided by an embodiment of the present invention;

[0058] Figure 3 A flowchart schematically illustrating an implementation of a camera calibration method provided by an embodiment of the present invention;

[0059] Figure 4 Schematically shows a flowchart of the implementation of the data set construction provided by an embodiment of the present invention;

[0060] Figure 5 A flowchart schematically illustrates an implementation process of aligning a camera head with the center of a target code according to an embodiment of the present invention;

[0061] Figure 6 A flowchart schematically illustrates the implementation of the process of identifying a target code and detecting a target identification result provided by an embodiment of the present invention;

[0062] Figure 7A flowchart schematically illustrates an implementation process of an assembly part fine positioning process based on composite target code technology provided by an embodiment of the present invention;

[0063] Figure 8 A schematic diagram schematically showing the rotation of a target code provided by an embodiment of the present invention along the x-axis direction;

[0064] Figure 9 A schematic diagram schematically showing the rotation of a target code provided by an embodiment of the present invention along the y-axis direction;

[0065] Figure 10 The figure schematically shows the rotation of the target code provided by the embodiment of the present invention along the z-axis direction. DETAILED DESCRIPTION

[0066] The description of the embodiments in this specification should be combined with the corresponding drawings, which should be considered a complete part of this specification. In the drawings, the shapes and thicknesses of the embodiments may be exaggerated and indicated for simplicity or convenience. Furthermore, the various structural components in the drawings will be described separately. It is worth noting that components not shown in the drawings or not described in words are known to those of ordinary skill in the art.

[0067] The description of the embodiments herein and any references to directions and orientations are for ease of description only and are not to be construed as limiting the scope of the present invention. The following description of the preferred embodiments may involve combinations of features, which may exist independently or in combination. The present invention is not specifically limited to the preferred embodiments. The scope of the present invention is defined by the claims.

[0068] According to the concept of the present invention, this embodiment discloses a method for identifying and positioning spacecraft assemblies based on target detection and composite target codes. By introducing a target detection algorithm and a new composite target code technology, the recognition and positioning of spacecraft assemblies are jointly realized, thereby improving the accuracy and stability of spacecraft assembly. First, the camera is calibrated using Zhang Zhengyou's camera calibration method, and then the improved YOLOv3 target detection algorithm is used to perform target detection, locate and identify the target assembly, and roughly estimate the pose in the standard coordinate system based on the pose of the target assembly in the image. Move the camera so that its visual center is aligned with the target center, identify the information in the target code, query the assembly information corresponding to the target code in the database, and verify the target detection and recognition results. After confirming that the target detection and recognition results are correct, the assembly is precisely positioned according to the target code. As Figure 1 As shown, the method specifically includes the following steps:

[0069] Step S101: calibrate the camera using the Zhang Zhengyou camera calibration method to correct the captured image.

[0070] In this embodiment, a camera is required to acquire an image. However, due to the influence of factors such as the three-dimensional spatial position relationship between the photographed object and the camera, lens distortion, and camera quality, the imaging system is inconsistent with the ideal pinhole model, and the captured image has geometric distortion. Especially when using a wide-angle lens, radial distortion will seriously affect the image quality, bringing difficulties to image processing. Therefore, it is necessary to correct the geometrically distorted image or calibrate the camera. The Zhang Zhengyou camera calibration method is used to calibrate the camera's internal parameters. The Zhang Zhengyou method is based on a single-plane checkerboard grid, which overcomes the disadvantage of traditional calibration methods requiring high-precision calibration objects. At the same time, it improves the accuracy and facilitates operation compared to self-calibration.

[0071] The camera imaging principle can be described by the following formula:

[0072] x=K[R|t]X

[0073] Among them, x is the coordinate in the camera, X is the real world coordinate, K is the intrinsic parameter matrix, and [R|t] is the extrinsic parameter matrix. Therefore, it can be written as the following formula:

[0074]

[0075] Where f is the focal length, s is the distortion parameter, (x0, y0) is the center point coordinate, and α is the scale parameter. The camera calibration process is the process of solving the internal and external parameter matrix.

[0076] In one embodiment, if Figure 3 As shown, the specific implementation process of calibrating the camera using the Zhang Zhengyou camera calibration method in step S101 and correcting the captured images includes: printing and securing a planar calibration plate composed of a checkerboard grid. For example, the planar calibration plate has 10×10 intersection points and can be secured to a non-deformable object. The planar calibration plate is placed in different positions and captured using a camera to capture images in different poses, for example, 12 images at different poses. The features of the planar calibration plate are detected to obtain the camera's intrinsic and extrinsic parameters and distortion parameters, and an optimized estimation of imaging accuracy is performed to improve the estimation accuracy.

[0077] Step S102: Using the improved YOLOv3 target detection algorithm, the target assembly in the image is identified and roughly located.

[0078] In one embodiment, if Figure 4As shown, the specific implementation process of using the improved YOLOv3 target detection algorithm to identify and roughly locate the target assembly in the image in step S102 includes: constructing a data set of assembly images, which is divided into a training set and a test set; constructing an improved YOLOv3 target detection network; using the training set to train the improved YOLOv3 target detection network, iterating until the loss function no longer decreases, and obtaining a weight file; using the weight file to test the images in the test set, identify the target assembly, and roughly estimate the position and posture of the target assembly in the standard coordinate system.

[0079] Before using YOLOv3 for target detection, it is necessary to first build a dataset. The dataset used in this embodiment is based on the automatic assembly system of spacecraft parts.

[0080] Specifically, the specific implementation process of constructing the dataset of assembly images in the above steps includes: collecting assembly images containing different workpiece categories and different shooting distances, such as collecting spacecraft assembly images, a total of 45 original images, including 5 types of workpieces, 3 shooting distances, a total of 15 categories, and three images of each category. The original images need to be preprocessed before they can be used. The assembly images are preprocessed by data format conversion and proportional scaling to form a basic dataset. The preprocessed images are represented as a 416×416×3 matrix, which is suitable for the input of the YOLO network. Due to the characteristics of spacecraft assemblies such as large size, large variety and small batch size, and easy collision damage, the images in the basic dataset may have problems such as occlusion and incomplete display. Therefore, data enhancement of the images in the basic dataset can enhance the recognition ability of the network and improve the stability of the network. Then, the images in the basic dataset are expanded through image rotation operations, and data enhancement is performed in combination with the proximity interpolation method to obtain an enhanced dataset. For example, an image is rotated at a random angle. Each image in the base dataset is rotated five times, expanding the original image into six images. Nearest interpolation is used to fill any gaps, resulting in an enhanced dataset containing 270 images. The images in this enhanced dataset are then labeled, recording their starting point coordinates, length, width, and classification information to obtain a labeled enhanced dataset. The labeled enhanced dataset is then divided into a training set and a test set. To determine the algorithm's prior box size, k-means clustering is used to classify the existing prior box sizes, resulting in nine prior box sizes at three resolutions. The prior box sizes for 32x downsampling are 323×340, 283×259, and 200×237, respectively; for 16x downsampling, they are 249×175, 166×181, and 149×140, respectively; and for 8x downsampling, they are 117×116, 89×94, and 57×58, respectively.

[0081] Specifically, in the specific implementation process of constructing the improved YOLOv3 target detection network in the above steps, GIOU is used instead of the interaction ratio (IOU) as an indicator to measure the target detection effect, and the Mish activation function is used instead of the ReLU activation function.

[0082] For example, the input is an RGB image with a resolution of 416*416, and the output results are the results of three search grids of size 13*13, 26*26, and 52*52. Each search grid includes the search results of three sampling boxes of different sizes, consisting of the coordinates of the four corners of the bounding box, the confidence of the bounding box, and the number of object categories (corresponding to four types of assembly models and a null class). Each search grid contains a total of 3×(4+1+5)=30 output results. The confidence is expressed as follows:

[0083]

[0084] Among them, P r (object) determines whether the target exists, the value is 1 or 0; The intersection-over-union ratio (IOR) indicates the degree of overlap between the true area and the predicted box. The larger the value, the better the detection effect. The calculation formula for this value is as follows:

[0085]

[0086] Here, area(pred) represents the area of ​​the detection box, and area(truth) represents the true area. IOU has the advantages of being non-negative and independent of scale. However, the calculation premise requires that the bounding box to be found and the true box are adjacent. If they are not adjacent, the size cannot be measured. At the same time, the relationship between the bounding box and the true box cannot be determined during the calculation process, such as whether the orientation is consistent. In particular, when the target object and the detection box are at a certain angle, detection and recognition cannot be performed. Therefore, GIOU is used instead of IOU. The formula of GIOU is as follows:

[0087]

[0088] Among them, A C It represents the minimum enclosed area of ​​the predicted box and the real box, that is, the area of ​​the minimum box that contains both the predicted box and the real box. U represents the union of the predicted box and the real box.

[0089] GIOU has good properties and operability, and its main features are as follows:

[0090] (1) Similar to the loss function calculated by the IOU method, the loss function of GIOU can be expressed as:

[0091]

[0092] (2) GIOU is insensitive to feature maps of different scales;

[0093] (3) The value range of GIOU is [-1, 1]. When the distance between the two boxes is infinite, the value of GIOU is -1; when the two rectangles overlap infinitely, the value of GIOU is 1, in which case GIOU = IOU.

[0094] (4) Unlike IOU which only focuses on overlapping areas, GIOU not only focuses on overlapping areas, but also on other non-overlapping areas, which can better reflect the degree of overlap between the two.

[0095] The main purpose of the activation function is to nonlinearize the input signal and is responsible for mapping the neuron input to the output. The Mish activation function is used instead of the ReLU activation function. The formula of the Mish activation function is as follows:

[0096] f(x)=x*tanh(ln(1+e x ))

[0097] Compared to the ReLU activation function, the Mish activation function can have infinite positive values, avoiding saturation due to capping, and slight changes in negative values ​​can have better gradient flow. This smooth activation function can express deeper information and better propagate information, thereby improving accuracy.

[0098] Step S103: Move the camera so that the visual center is aligned with the center of the target code of the target assembly.

[0099] In one embodiment, if Figure 5 As shown, the specific implementation process of moving the camera in step S103 so that the visual center is aligned with the center of the target code of the target assembly includes: using a visual sensor to collect visual information around the robotic arm and determine the posture information; using a force protection sensor to generate feedback information when the robotic arm collides; performing obstacle detection, target recognition and posture analysis based on the posture information and feedback information, and making a decision; and controlling the robotic arm to move the camera to a specified position based on the decision information so that the visual center is aligned with the center of the target code of the target assembly.

[0100] Specifically, after obtaining the assembly's pose using the YOLOv3 object detection algorithm, sensors are used to collect surrounding information. The sensor component consists of a vision sensor and a force protection sensor. The vision sensor collects visual information about the machine's surroundings for use in determining its pose, while the force protection sensor provides feedback when the robot arm collides, preventing damage caused by collisions. The sensors then transmit this collected environmental information to a host computer. The analysis and decision-making system uses the data collected by the sensors to perform operations such as obstacle detection, object recognition, and pose analysis on the host computer, making relevant decisions and directing the robot's next steps. During this process, the communication system is responsible for establishing a communication interface between the robot, sensors, and the host computer. It transmits the surrounding information collected by the vision and force protection sensors to the host computer, and then transmits the control information generated by the host computer to the control system. The control system primarily controls the robot arm. Specifically, it requires the robot arm to reach a specified position. Upon receiving the host computer's decision information, it controls the robot arm to move to the corresponding position, ensuring that the camera is aligned with the center of the target code on the target assembly.

[0101] Step S104: Identify the target code, obtain corresponding assembly part information, and verify the recognition result of the target detection algorithm.

[0102] In one embodiment, if Figure 6 As shown, the specific implementation process of identifying the target code, obtaining the corresponding assembly information, and verifying the recognition result of the target detection algorithm in step S104 includes: using the composite target code technology to identify the information in the target code image captured by the camera; querying the corresponding assembly information in the database according to the index information of the information storage layer of the target code; comparing the assembly information with the recognition result of the target detection algorithm, and re-detecting the target assembly if the recognition result is incorrect.

[0103] like Figure 2 As shown, in this embodiment, the target code is a new type of composite target code technology, which specifically includes: (1) an auxiliary positioning layer, the bottom layer is a checkerboard calibration line, which can be used as a posture calibration plate to assist in the posture recognition of the assembly; (2) an information storage layer, the middle layer is a classic QR code, which records the information of the assembly and the precise position relationship between the target code and the assembly; (3) a personnel interaction layer, the top layer is an identification code, which records the equipment name, number and person in charge of the assembly, facilitating identification and safety management of operators; (4) a center mark point, with a center mark point in the center for precise positioning.

[0104] The database in the above steps records the index number, name, current assembly progress, historical assembly records, complete assembly process and other information of the assembly. After identifying the information contained in the target code, the assembly corresponding to the target code information is queried in the database based on the index information recorded in the information storage layer to determine the precise positional relationship between the assembly information and the assembly. In order to facilitate information management, only the administrator can enter and modify equipment information in this database. The database system can not only provide data query, but also perform quality control, knowledge management and data tracking on assemblies at the same time. Since the recording capacity of a single target code is limited, the target code is only used as an index. After the system identifies the information contained in the target code, it can find the assembly corresponding to the information contained. The assembly number is "DF-assembly category number-assembly number".

[0105] Step S105: If the recognition result is correct, the target assembly is precisely positioned according to the target code using composite target code technology.

[0106] In one embodiment, if Figure 7 As shown, the specific implementation process of using the composite target code technology to finely locate the target assembly according to the target code in step S105 includes: calculating the distance between the camera and the center point of the target code, and establishing a target code coordinate system based on the target code image captured by the camera; extracting corner points in the target code image using the Harris corner detection algorithm; calculating the rotation angle of the target code image in space based on the corner points; determining the precise positional relationship between the assembly information and the target assembly based on the assembly information corresponding to the target code, and determining the precise positions of the corner points and the rotation angle in the target code coordinate system; and calculating the precise position of the corner points in the standard coordinate system based on the rotation angle.

[0107] Specifically, the specific implementation process of calculating the distance h between the camera and the center point of the target code in the above steps and establishing the target code coordinate system based on the target code image captured by the camera includes:

[0108] Coordinate origin: The center mark point of the target code image is the origin of the coordinate system;

[0109] x-axis: The horizontal axis in the target code image is the x-axis of the coordinate system, with the right direction being the positive direction;

[0110] Y-axis: The vertical axis in the target code image is the y-axis of the coordinate system, with upward being the positive direction;

[0111] z-axis: The axis passing through the center point of the target code image and perpendicular to the x-axis and y-axis is the z-axis of the coordinate system, and the direction toward the camera is the positive direction.

[0112] For example, using (x i,j,y i,j ,z i,j ) represents the coordinates of the target code image point in the coordinate system; use (x' i,j ,y' i,j ,z' i,j ) represents the coordinates of the actual position point of the target code in this coordinate system; distance is used to represent the actual distance between two adjacent corner points in the target code.

[0113] Specifically, the specific implementation process of extracting corner points in the target code image using the Harris corner detection algorithm in the above steps includes:

[0114] (1) Calculate the horizontal and vertical gradients of the pixel points of the target code image and the product of the two to generate a matrix M:

[0115]

[0116] Among them, I x and I y are the gradients in the horizontal and vertical directions, I xy =I x I y ,

[0117] (2) Gaussian filtering is performed on the target code image to obtain a new matrix M; the discrete two-dimensional zero-mean Gaussian function is:

[0118]

[0119] (3) Calculate the interest value of each pixel corresponding to the original target code image, that is, the R value:

[0120]

[0121] Here, k is an empirical constant, and in this embodiment, k=0.04.

[0122] (4) Select the pixel point corresponding to the maximum interest value in the local range;

[0123] (5) Set the threshold and select the corner point (x i,j ,y i,j ,0), where i is the i-th corner point in the horizontal direction and j is the j-th corner point in the vertical direction. Since the image is a two-dimensional plane, the z-axis coordinates of all points on the image are 0. At the same time, since the number of corner points in each row of the auxiliary positioning layer is not the same, for the sake of uniform description, the number of corner points in the horizontal direction is set to n and the number of corner points in the vertical direction is set to m. Here, m and n are not fixed values, but vary according to the actual number of corner points.

[0124] Specifically, such as Figure 8-10 As shown, the specific implementation process of calculating the rotation angle of the target code image in space according to the corner points in the above steps includes:

[0125] like Figure 8 As shown, the rotation angle α along the x-axis is calculated using two adjacent vertical corner points. The calculation formula is as follows:

[0126]

[0127] To make the angle calculation more accurate, the average value of the distances between adjacent corner points and the distance between the first and last corner points is used as the basis for calculating the α angle of a row. The average value of the α angles of all rows is then taken as the actual α value. The final calculation formula is as follows:

[0128]

[0129] like Figure 9 As shown, two adjacent corner points in the horizontal direction are used to calculate the rotation angle β along the y-axis. The calculation formula is as follows:

[0130]

[0131] To make the angle calculation more accurate, the average value of the distances between adjacent corner points and the distance between the first and last corner points is used as the basis for calculating the β angle of a column. The average value of the β angles of all columns is then taken as the actual β value. The final calculation formula is as follows:

[0132]

[0133] like Figure 10 As shown, two adjacent corner points in the horizontal direction are used to calculate the rotation angle θ along the z-axis. The calculation formula is as follows:

[0134]

[0135] At the same time, the two adjacent vertical corner points are used to calculate the rotation angle θ along the z-axis. The calculation formula is as follows:

[0136]

[0137] To make the angle calculation more accurate, the average value of the vertical distances between adjacent corner points and the distance between the first and last corner points are used as the basis for calculating the θ angle of a row. The average value of the horizontal distances between adjacent corner points and the distance between the first and last corner points are used as the basis for calculating the θ angle of a column. The average value of all row and column angles is then taken as the actual θ value. The final calculation formula is as follows:

[0138]

[0139] Since the spatial motion of an object includes six degrees of freedom, namely three movements along the x-axis, y-axis, and z-axis, and three rotations along the x-axis, y-axis, and z-axis, and since the camera has been aligned with the center point of the target code, there is no movement in the x-axis and y-axis directions for the target code image. At the same time, the movement in the z-axis direction is mainly used to calculate the distance h between the camera and the center point of the target code. Therefore, only the rotation angles of the target code image along the x-axis, y-axis, and z-axis directions need to be calculated.

[0140] Specifically, the specific calculation process of calculating the precise position of the corner point in the standard coordinate system according to the rotation angle in the above steps includes the following formula:

[0141]

[0142] Among them, the coordinates of the exact position are (x i ' ,j ,y i ' ,j ,z i ' ,j ); the coordinates of the corner points are (x i,j ,y i,j ,0), i is the i-th corner point in the horizontal direction, j is the j-th corner point in the vertical direction; α represents the rotation angle of the longitudinal adjacent corner points of the target code image along the x-axis direction, β represents the rotation angle of the transverse adjacent corner points of the target code image along the y-axis direction, and θ represents the average value of the rotation angle of the adjacent corner points in all rows and columns of the target code image along the z-axis direction.

[0143] The serial numbers of the above-mentioned steps involved in the method of the present invention do not mean the order of execution of the method. The execution order of each step should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0144] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for identifying and locating spacecraft components based on target detection and composite target codes, comprising: Use Zhang Zhengyou's camera calibration method to calibrate the camera and correct the captured image; Use the improved YOLOv3 target detection algorithm to identify and roughly locate the target assembly in the image; The improved YOLOv3 target detection algorithm is used to identify and roughly locate the target assembly in the image, including: Construct a dataset of assembly image data, which is divided into training set and test set; Build an improved YOLOv3 target detection network; Use the training set to train the improved YOLOv3 target detection network, iterate until the loss function no longer decreases, and obtain the weight file; Use the weight file to test the images in the test set, identify the target assembly, and roughly estimate the pose of the target assembly in the standard coordinate system; Build an improved YOLOv3 target detection network, including using GIOU instead of interaction ratio as the metric for measuring target detection performance, and using Mish activation function instead of ReLU activation function; Move the camera so that the visual center is aligned with the center of the target code of the target assembly; Identify the target code, obtain corresponding assembly information, and verify the recognition result of the target detection algorithm; If the recognition result is correct, the target assembly is precisely positioned according to the target code using composite target code technology; Using composite target code technology to precisely locate the target assembly according to the target code, including: Calculating the distance between the camera and the center point of the target code, and establishing a target code coordinate system based on the target code image captured by the camera; Extracting corner points in the target code image using Harris corner detection algorithm; Calculating the rotation angle of the target code image in space according to the corner points; Determining the precise positional relationship between the assembly information and the target assembly based on the assembly information corresponding to the target code, and determining the precise positions of the corner point and the rotation angle in the target code coordinate system; The precise position of the corner point in the standard coordinate system is calculated according to the rotation angle.

2. The method according to claim 1, characterized in that Use Zhang Zhengyou's camera calibration method to calibrate the camera and correct the captured images, including: Print a flat calibration plate consisting of a checkerboard and fix it; Placing the plane calibration plate at different positions and using a camera to capture images of different postures; The features of the plane calibration plate are detected to obtain the internal and external parameters and distortion parameters of the camera, and to optimize and estimate the imaging accuracy.

3. The method according to claim 1, characterized in that Construct a dataset of assembly images, including: Collect assembly images containing different workpiece types and different shooting distances; Performing data format conversion and proportional scaling preprocessing on the assembly part images to form a basic data set; Expanding the image data of the images in the basic data set by performing an image rotation operation, and performing data enhancement in combination with a proximity interpolation method to obtain an enhanced data set; The images in the enhanced data set are marked, and the starting point coordinates, length, width and classification information of the images are recorded to obtain a marked enhanced data set.

4. The method according to claim 1, wherein Moving the camera so that the visual center is aligned with the center of the target code of the target assembly, including: Use visual sensors to collect visual information around the robotic arm and determine its position and posture information; Use force protection sensors to generate feedback information when the robot arm collides; Perform obstacle detection, target recognition, and posture analysis based on posture information and feedback information, and make decisions; The robot arm is controlled to move the camera to a specified position according to the decision information, so that the visual center is aligned with the center of the target code of the target assembly.

5. The method according to claim 1, wherein Identify the target code, obtain the corresponding assembly part information, and verify the target detection recognition result, including: Using composite target code technology to identify information in the target code image captured by the camera; According to the index information of the information storage layer of the target code, query the database for corresponding assembly information; The assembly information is compared with the recognition result of the target detection algorithm. If the recognition result is wrong, the target assembly is re-detected.

6. The method according to claim 1, characterized in that The Harris corner detection algorithm is used to extract corner points in the target code image, including: Calculate the horizontal and vertical gradients of the pixels of the target code image and their product to generate a matrix: Performing Gaussian filtering on the target code image to obtain a new matrix; Calculate the interest value of each pixel corresponding to the original target code image; Select the pixel point corresponding to the local maximum interest value; Set the threshold and select the corner point (x i,j ,y i,j ,0), where i is the i-th corner point in the horizontal direction and j is the j-th corner point in the vertical direction.

7. The method according to claim 1, characterized in that The exact position of the corner point in the standard coordinate system is calculated according to the rotation angle. The specific calculation formula is as follows: Among them, the coordinates of the exact position are (x i ',j,y i ',j,z i ',j); the coordinates of the corner point are (x i,j ,y i,j ,0), i is the i-th corner point in the horizontal direction, j is the j-th corner point in the vertical direction; α represents the rotation angle of the longitudinal adjacent corner points of the target code image along the x-axis direction, β represents the rotation angle of the transverse adjacent corner points of the target code image along the y-axis direction, and θ represents the average value of the rotation angle of the adjacent corner points in all rows and columns of the target code image along the z-axis direction.

Images