A multi-camera small target identification and joint positioning method and system

Abstract

A multi-camera small-size object recognition and positioning method and system, the method comprising: S1: camera external parameter calibration S2: small target visual detection S3: small target point cloud background filtering S4: joint positioning of two cameras S5: target center position estimation compensation. The application provides target information observed under multiple perspectives through the simultaneous work of multiple cameras, and further improves the robustness and accuracy of small-size target recognition and positioning through convolutional neural network, point cloud filtering, spatial clustering and other means.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine vision, in particular to a multi-camera small target recognition and joint positioning method and system. BACKGROUND

[0002] Multi-camera small target recognition and joint positioning is an important research direction in the field of computer vision. In practical applications, such as video surveillance, autonomous driving and other scenarios, accurate recognition and positioning of small targets are required for subsequent processing and decision-making. However, in complex environments, a single camera often cannot complete the task of accurate positioning and recognition of small targets, so multiple cameras are needed for joint processing.

[0003] The research of multi-camera small target recognition and joint positioning faces some challenges. First, due to the differences in viewing angle and lighting conditions between different cameras, the generated images have large differences, which will affect the performance of the model. Second, the size and shape of small targets are diverse, plus noise and occlusion interference, making it more difficult to recognize and locate small targets. In addition, how to register and fuse data between cameras also needs to be solved.

[0004] In order to solve the above problems, researchers have proposed a series of methods to realize multi-camera small target recognition and joint positioning. For example, some researchers use multi-task learning methods to simultaneously process classification and positioning of small targets to improve model accuracy, but these two methods are difficult to label and require a lot of time for data labeling. Other researchers use deep learning and other technologies for feature extraction and matching to achieve multi-camera data registration and fusion, but this method has the problem of difficulty in transferring due to changes in camera internal and external parameters. SUMMARY

[0005] The present application overcomes the above-mentioned shortcomings of the prior art and provides a method for recognizing and positioning small targets through multiple cameras of a robot.

[0006] A multi-camera small target recognition and joint positioning method of the present application comprises

[0007] S1: Camera external parameter calibration;

[0008] S2: Detect small target vision;

[0009] S3: Filter the point cloud background of small targets;

[0010] S4: Joint positioning of two cameras;

[0011] S5: Estimate the compensation target center position.

[0012] S1 specifically comprises:

[0013] The depth is aligned to the RGB image output, and the RGB camera of the right eye is aligned to the RGB camera of the left eye through the external parameter calibration.

[0014] The calibration process is as follows:

[0015] First, the external parameters between the left eye and the right eye are roughly calibrated using a calibration board; then, the first calibrated value is used as the initial value of ICP, and the point cloud in the overlapping interval is further used to perform secondary calibration through the ICP matching algorithm combined with color, to obtain more accurate calibration parameters.

[0016] Among them, in order to make the ICP matching obtain good results, the reference objects with obvious features are arranged in the overlapping zone.

[0017] Further, the reference objects are several color boxes with obvious features arranged from low to high.

[0018] Among them, step S2 is specifically:

[0019] First, when the camera configuration is performed, a configuration with larger pixels such as 1280*720 is selected as the output photo size; at the same time, when the photo is input into the convolutional network, a sliding window is used for sampling; the input size of the neural network model yolox used is 640*640, and the sliding window is divided.

[0020] Among them, step S3 specifically includes:

[0021] The following method is used for background filtering:

[0022] Suppose the target point cloud obj_points, the data format is a three-dimensional matrix, and the matrix size is [a, b, 3], a and b are the length and width of the detection box, and 3 is the value of the x (horizontal direction of the camera), y (vertical direction of the camera), and z (direction away from the camera) of the point cloud. The background filtering method is to first calculate the median of the third channel of the point cloud, and then delete the point cloud 1cm away from the z direction center, described in python:

[0023] zs = obj_points[:, 2] (9)

[0024] z = np.median(zs) (10)

[0025] obj_points = np.delete(obj_points, np.where((zs < z - 0.01) | (zs > z+ 0.01)), axis=0) (11).

[0026] Among them, step S4 is specifically:

[0027] After obtaining the target point cloud of two cameras, the point clouds of the same target are merged, the point cloud is observed from multiple directions to obtain a more comprehensive description of the object, and the center of the object is further calculated based on the merged point cloud.

[0028] First, the envelope box of the target under a single camera is calculated, and the center of the box represents the center of the target. The target centers within 1 cm are judged as the same target, and the point clouds are merged based on this judgment.

[0029] The step S5 specifically comprises:

[0030] The basis for compensating the target center position is that the closer the target, the more top surface can be seen, the more accurate the center estimate, and the farther the target, only the front surface can be seen, and the estimated center is close to the target. According to the width of the target, the far target is compensated by a certain percentage in the far direction, and the maximum compensation amount is 1 / 3 of the target width. The target position is compensated by this method. Assuming that the target position is:

[0031] P=[x,y,z] (1)

[0032] Where x is ahead, y is on the side, and z is upward.

[0033] The target size is:

[0034] Size=[d1,d2,d3] (2)

[0035] First, the x direction is compensated:

[0036] x_offset=(e-e^(0.4-x)) / e*d2 / 3 (3)

[0037] Assuming that the target size d2=0.03, the compensation value x_offset curve has a large slope in the near place and a small slope in the far place, and approaches 1 / 3 of the target size at infinity.

[0038] Second, the y direction is compensated:

[0039] y_offset=y / |y|*(1-e^(-|y|)) / e*d2 (4)

[0040] The compensation value y_offset curve has a large slope in the place with small absolute value and a small slope in the place with large absolute value, and the compensation amount is about 7mm at y direction ±1m.

[0041] x1=x+x_offset (5)

[0042] y1=y+y_offset (6)

[0043] z1=z (7)

[0044] The compensated target center position P1 is:

[0045] P1=[x1, y1, z1] (8)

[0046] Where e is a natural constant.

[0047] A second aspect of the application relates to a multi-camera small target recognition and joint positioning system, comprising

[0048] An inter-camera extrinsic parameter calibration module for calibrating the inter-camera extrinsic parameters;

[0049] A small target visual detection module for detecting small target vision;

[0050] A small target point cloud background filtering module for detecting the point cloud background of the small target;

[0051] A two-camera joint positioning module for jointly positioning the two cameras;

[0052] A target center position estimation compensation module for estimating and compensating the target center position.

[0053] A third aspect of the application relates to a computer readable storage medium having a program stored thereon, wherein the program is executed by a processor to implement a multi-camera small target recognition and joint positioning method of the application.

[0054] A fourth aspect of the application relates to a computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement a multi-camera small target recognition and joint positioning method of the application.

[0055] The innovation of the application is:

[0056] In the camera extrinsic parameter calibration, coarse registration based on a calibration board and fine registration by icp are adopted to solve the problem of low annotation accuracy caused by small camera overlapping angle; in the target recognition, a 2D detection and background point cloud filtering method instead of semantic segmentation is adopted to reduce the difficulty of small target annotation; by improving the photo resolution, using sliding window sampling, selecting a large size input neural network, etc., the small target recognition accuracy of the convolutional neural network is improved; by providing different position observation view points of the point cloud through multiple cameras, more accurate object information is provided; and by the compensation algorithm, the accuracy of the camera position estimation is improved.

[0057] The advantages of the application are:

[0058] The small target object recognition has the advantages of small implementation difficulty, high recognition and positioning accuracy, and good scalability on different machine platforms. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 This is a flowchart of the method of the present invention.

[0060] Figure 2 This is a structural schematic diagram of the robot that applies the present invention.

[0061] Figure 3 This is a schematic diagram showing the view from the front-facing camera of the robot using this invention.

[0062] Figure 4 This is a schematic diagram showing the position of the calibration plate of the robot using the present invention.

[0063] Figure 5 This is a schematic diagram of the sliding sampling window of the robot using the present invention.

[0064] Figure 6 This is a schematic diagram of point cloud merging according to the present invention.

[0065] Figure 7 This is a schematic diagram of the x-direction compensation value of the present invention.

[0066] Figure 8 This is a schematic diagram of the y-direction compensation value of the present invention.

[0067] Figure 9 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0070] Example 1

[0071] Figure 1 This is a flowchart illustrating a multi-camera small target recognition and joint localization method according to an embodiment of the present invention. Figure 1 As shown, a multi-camera small target recognition and joint localization method in an embodiment of the present invention may include the following steps:

[0072] This method uses a hexapod robot as an example, but it is not limited to this type of robot. For example... Figure 2As shown, there are left and right eye cameras in front of the robot, and the application realizes small target positioning based on the two cameras, and the same method can be extended to more cameras. The specific implementation is as follows:

[0073] S1: Camera extrinsic calibration

[0074] S2: Small target visual detection

[0075] S3: Point cloud background filtering of small target

[0076] S4: Joint positioning of two cameras

[0077] S5: Target center position estimation compensation

[0078] 1. Camera extrinsic calibration

[0079] Two front cameras use depth cameras, which can output RGB images and depth images. The RGB image can be aligned to the depth image output, or the depth image can be aligned to the RGB image output. However, since the depth map has many null values, aligning the RGB image to the depth image output will cause many gaps in the RGB image, affecting subsequent target recognition. Therefore, the depth is aligned to the RGB image output, and the RGB camera of the right eye is further aligned to the RGB camera of the left eye through extrinsic calibration.

[0080] As shown in Figure 3 , the blue area is the view angle of the left eye camera, the yellow area is the view angle of the right eye camera, and the middle is the overlap interval. When calibrating, place a calibration board in the overlap area, as shown in Figure 4 , since the overlap area is small, only a small calibration board can be used, so the calibration accuracy is low; if ICP matching is used directly, it will be difficult to obtain good calibration results due to the large difference in the initial point cloud angle. Therefore, the following scheme is used for calibration in this embodiment:

[0081] First, use the calibration board to roughly calibrate the extrinsic parameters between the left and right eyes. Then, use the first calibration value as the initial value of ICP, and further use the point cloud in the overlap interval to perform secondary calibration through the ICP matching algorithm combined with color, to obtain more accurate calibration parameters. In order to obtain good results of ICP matching, it is necessary to arrange reference objects with obvious features in the overlap zone, such as arranging several color boxes with obvious features from low to high.

[0082] 2. Small target visual detection

[0083] This invention identifies small-sized targets. Due to their small size, these targets have few features within the photograph, hindering visual recognition. To address this issue, this method first selects a larger pixel configuration, such as 1280*720, as the output photo size during camera configuration. Simultaneously, when inputting the photo into the convolutional network, it does not perform photo size normalization scaling but instead uses a sliding window sampling method. The neural network model used in this invention has an input size of 640*640 (YOLOv). Furthermore, considering that the target object will not appear in the upper half of the photo, the segmentation sliding window is as follows... Figure 4 As shown. Ultimately, by increasing the target's resolution, the recognition accuracy is improved.

[0084] 3. Point cloud background filtering for small targets

[0085] This invention employs 2D detection for target recognition, without semantic segmentation, because the target has few pixels, making semantic segmentation annotation difficult. However, using 2D detection also introduces a problem: when using point clouds for target localization, background point clouds within the detection bounding box need to be removed. Since the target and the ground are connected, background filtering presents some challenges. This invention uses the following method for background filtering:

[0086] Assuming the target point cloud is obj_points, the data format is a 3D matrix with dimensions [a, b, 3], where a and b are the length and width of the bounding box, and 3 represents the x (horizontal direction), y (vertical direction), and z (away from camera) values ​​of the point cloud. The background filtering method first calculates the median of the third channel of the point cloud, and then deletes points within 1 cm before and after the median in the z direction. This is described using Python.

[0087] zs = obj_points[:,2] (9)

[0088] z = np.median(zs) (10)

[0089] obj_points=np.delete(obj_points,np.where((zs<z-0.01)|(zs> z+0.01),axis=0) (11)

[0090] 4. Joint positioning of two cameras

[0091] After obtaining the target point clouds from two cameras, it is necessary to merge the point clouds of the same target. By observing the point clouds from multiple angles, a richer description of the object can be obtained, and the object's grasping center can be further calculated from the merged point cloud. For example... Figure 6 As shown, the blue point cloud represents the target point cloud captured by the right eye camera, while the yellow point cloud represents the point cloud captured by the left eye camera. It is necessary to determine whether the point clouds from both cameras represent the same target in order to merge them.

[0092] First, the bounding box of the target under a single camera is calculated, and the center of the bounding box is used to represent the center of the target. Then, the target centers are spatially clustered, with target centers within 1 cm considered as the same target. This criterion is used to merge the point clouds. Compared to clustering all point clouds, this reduces the computational cost, and...

[0093] When the target proximity is small, it exhibits better merging performance.

[0094] 5. Target center location estimation compensation

[0095] Since the front-facing camera can only observe the front of the target, estimating the target center based on the point cloud from the front will introduce some error. Therefore, compensation is needed to correct the target center position. The compensation is based on the principle that closer targets show more of the top surface, resulting in a more accurate center estimation; farther targets only show the front, leading to a closer estimated center. Furthermore, based on the target's width, a certain percentage is compensated for farther targets in the direction of distance, with the maximum compensation being applied here.

[0096] The measurement is 1 / 3 of the target width, and position compensation is performed using this method. Assume the target position is:

[0097] P = [x, y, z] (1)

[0098] In this diagram, x is forward, y is to the side, and z is upward.

[0099] The target size is:

[0100] Size=[d1,d2,d3] (2)

[0101] First, compensation is performed in the x-direction:

[0102] x_offset=(ee^(0.4-x)) / e*d2 / 3 (3)

[0103] Assuming the target size d2 = 0.03, the compensation value x_offset curve is as follows: Figure 7 As shown. The characteristics of the compensation curve are: the slope is large near the target and small far away, and it approaches 1 / 3 of the target size at infinity.

[0104] Further compensation is applied in the y-direction:

[0105] y_offset=y / |y|*(1-e^(-|y|)) / e*d2 (4)

[0106] The compensation value y_offset curve is as follows Figure 8 As shown, the characteristics of the compensation curve are: the slope is large where the absolute value is small and the slope is small where the absolute value is large. The compensation amount is about 7mm at ±1 meter in the y direction.

[0107] x1 = x + x_offset (5)

[0108] y1=y+y_offset (6)

[0109] z1=z (7)

[0110] The compensated target center position P1 is:

[0111] P1=[x1,y1,z1] (8)

[0112] Where e is the natural constant, approximately equal to 2.71828.

[0113] Example 2

[0114] like Figure 9 This embodiment provides a multi-camera small target recognition and joint localization system to implement the multi-camera small target recognition and joint localization method described in Embodiment 1, including...

[0115] Camera extrinsic calibration module, used to calibrate extrinsic parameters between cameras;

[0116] The small target visual detection module is used to detect small targets visually.

[0117] The point cloud background filtering module for small targets is used to detect the point cloud background of small targets.

[0118] A joint positioning module for two cameras, used for joint positioning of two cameras;

[0119] The target center location estimation and compensation module is used to estimate and compensate the target center location.

[0120] Example 3

[0121] This embodiment relates to a computer-readable storage medium storing a program that, when executed by a processor, implements a multi-camera small target recognition and joint localization method of Embodiment 1.

[0122] Example 4

[0123] This embodiment relates to a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a multi-camera small target recognition and joint localization method of Embodiment 1.

[0124] At the hardware level, the computing device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the method of Embodiment 1. Of course, in addition to the software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0125] Improvements in a technology can be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many improvements to the methodology can now be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that an improvement in methodology cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog are the most commonly used. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0126] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0127] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0128] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, in implementing this invention, the functions of each unit can be implemented in one or more software and / or hardware components.

[0129] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0131] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0132] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0133] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0134] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0135] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0136] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0137] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0138] This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0139] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0140] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A multi-camera method for small target recognition and joint localization, characterized in that, include S1: External parameter calibration between cameras; Specifically, it includes: Choose to align the depth to the RGB image output, and align the right target RGB camera to the left target RGB camera using extrinsic calibration; The calibration process is as follows: First, the extrinsic parameters between the left and right eyes are roughly calibrated using a calibration board; then, the values ​​from the first calibration are used as the initial values ​​for ICP, and a second calibration is performed using the point cloud of the overlapping area by combining the color with the ICP matching algorithm to obtain more accurate calibration parameters. S2: Visual detection of small targets; S3: Filter the point cloud background of small targets; specifically including: Background filtering is performed using the following methods: Assuming the target point cloud is obj_points, the data format is a 3D matrix with dimensions [a, b, 3], where a and b are the length and width of the detection box, and 3 represents the values ​​in the x, y, and z directions of the point cloud. The background filtering method first calculates the median of the third channel of the point cloud, and then deletes points within 1 cm before and after the median in the z direction. This is described using Python. zs = obj_points[:, 2] (9) z = np.median(zs) (10) obj_points=np.delete(obj_points, np.where((zs < z - 0.01) | (zs > z +0.01)),axis=0) (11) S4: Joint multi-camera positioning for small targets; S5: Estimate the location of the compensation target center; specifically including: The basis for compensating for the target's center position is that closer targets show more of their top surface, resulting in a more accurate center estimation; farther targets only show their front surface, leading to a closer estimated center. Furthermore, based on the target's width, farther targets are compensated proportionally in the direction of distance, with a maximum compensation of 1 / 3 of the target's width. This is used for position compensation. Assuming the target position is: P=[x,y,z] (1) Where x faces forward, y is to the side, and z faces upward; The target size is: Size=[d1,d2,d3] (2) First, compensation is performed in the x-direction: x_offset=(ee^(0.4-x)) / e*d2 / 3 (3) Assuming the target size d2 = 0.03, the compensation value x_offset curve has a large slope near the target and a small slope far away, and it approaches 1 / 3 of the target size at infinity. Secondly, compensation is performed in the y-direction: y_offset=y / |y|*(1-e^(-|y|)) / e*d2 (4) The slope of the compensation value y_offset curve is large where the absolute value is small and small where the absolute value is large. The compensation amount is 7mm at ±1 meter in the y direction. x1 = x + x_offset (5) y1 = y + y_offset (6) z1=z (7) The compensated target center position P1 is: P1=[x1,y1,z1] (8) Where e is the natural constant.

2. The multi-camera small target recognition and joint localization method according to claim 1, characterized in that, To ensure good ICP matching results, reference objects are placed in the overlapping area.

3. The multi-camera small target recognition and joint localization method according to claim 2, characterized in that, The reference objects are several colored boxes arranged in a near-low and far-high order.

4. The multi-camera small target recognition and joint localization method according to claim 1, characterized in that, S2 specifically refers to: First, when configuring the camera, 1280*720 was selected as the output photo size; at the same time, when inputting the photo into the convolutional network, a sliding window sampling method was used; the input size of the neural network model YOLOx was 640*640, and a segmented sliding window was used.

5. The multi-camera small target recognition and joint localization method according to claim 1, characterized in that, Step S4 is as follows: After obtaining the target point clouds from two cameras, the point clouds of the same target are merged. By observing the point clouds from multiple angles, a richer description of the object is obtained, and the object's grasping center is further calculated from the merged point cloud. First, the bounding box of the target under a single camera is calculated, and the center of the box is used to represent the center of the target. Then, the centers of the targets are clustered in space to determine that the centers of targets within 1 cm are the same target. Based on this judgment, the point clouds are merged.

6. A multi-camera small target recognition and joint positioning system, characterized in that, include A camera-to-camera extrinsic parameter calibration module is used to calibrate the extrinsic parameters between cameras; specifically, it includes: Choose to align the depth to the RGB image output, and align the right target RGB camera to the left target RGB camera using extrinsic calibration; The calibration process is as follows: First, the extrinsic parameters between the left and right eyes are roughly calibrated using a calibration board; then, the values ​​from the first calibration are used as the initial values ​​for ICP, and a second calibration is performed using the point cloud of the overlapping area by combining the color with the ICP matching algorithm to obtain more accurate calibration parameters. The small target visual detection module is used to detect small targets visually. The point cloud background filtering module for small targets is used to detect the point cloud background of small targets; specifically, it includes: Background filtering is performed using the following methods: Assuming the target point cloud is obj_points, the data format is a 3D matrix with dimensions [a, b, 3], where a and b are the length and width of the detection box, and 3 represents the values ​​in the x, y, and z directions of the point cloud. The background filtering method first calculates the median of the third channel of the point cloud, and then deletes points within 1 cm before and after the median in the z direction. This is described using Python. zs = obj_points[:, 2] (9) z = np.median(zs) (10) obj_points=np.delete(obj_points, np.where((zs < z - 0.01) | (zs > z +0.01)),axis=0) (11) A joint positioning module for two cameras is used for joint positioning of small targets; The target center location estimation and compensation module is used to estimate and compensate for the target center location; specifically, it includes: The basis for compensating for the target's center position is that closer targets show more of their top surface, resulting in a more accurate center estimation; farther targets only show their front surface, leading to a closer estimated center. Furthermore, based on the target's width, farther targets are compensated proportionally in the direction of distance, with a maximum compensation of 1 / 3 of the target's width. This is used for position compensation. Assuming the target position is: P=[x,y,z] (1) Where x faces forward, y is to the side, and z faces upward; The target size is: Size=[d1,d2,d3] (2) First, compensation is performed in the x-direction: x_offset=(ee^(0.4-x)) / e*d2 / 3 (3) Assuming the target size d2 = 0.03, the compensation value x_offset curve has a large slope near the target and a small slope far away, and it approaches 1 / 3 of the target size at infinity. Secondly, compensation is performed in the y-direction: y_offset=y / |y|*(1-e^(-|y|)) / e*d2 (4) The slope of the compensation value y_offset curve is large where the absolute value is small and small where the absolute value is large. The compensation amount is 7mm at ±1 meter in the y direction. x1 = x + x_offset (5) y1 = y + y_offset (6) z1=z (7) The compensated target center position P1 is: P1=[x1,y1,z1] (8) Where e is the natural constant.

7. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the method of any one of claims 1-5.

8. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-5.

Images